Nov 03 01:17:23.804674 ip-10-0-136-16 systemd[1]: Starting Kubernetes Kubelet... Nov 03 01:17:24.173793 ip-10-0-136-16 kubenswrapper[2578]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 01:17:24.173793 ip-10-0-136-16 kubenswrapper[2578]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 03 01:17:24.173793 ip-10-0-136-16 kubenswrapper[2578]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 01:17:24.173793 ip-10-0-136-16 kubenswrapper[2578]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 01:17:24.177511 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.177429 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 01:17:24.177511 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.177512 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 01:17:24.177574 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.177517 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 01:17:24.177574 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.177521 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 01:17:24.177574 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.177524 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 01:17:24.177574 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.177527 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 01:17:24.177574 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.177530 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 01:17:24.177574 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.177533 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 01:17:24.177574 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.177536 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 01:17:24.177574 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.177538 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 01:17:24.177574 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.177541 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 01:17:24.177574 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.177544 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 01:17:24.177574 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.177547 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 01:17:24.177574 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.177549 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 01:17:24.177574 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.177552 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 01:17:24.177574 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.177555 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 01:17:24.177574 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.177557 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 01:17:24.177574 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.177560 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 01:17:24.177574 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.177563 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 01:17:24.177574 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.177571 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 01:17:24.177574 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.177574 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 01:17:24.177574 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.177577 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 01:17:24.178038 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.177580 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 01:17:24.178038 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.177583 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 01:17:24.178038 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.177585 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 01:17:24.178038 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.177588 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 03 01:17:24.178038 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.177591 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 01:17:24.178038 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.177593 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 01:17:24.178038 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.177596 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 01:17:24.178038 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.177598 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 01:17:24.178038 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.177601 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 01:17:24.178038 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.177603 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 01:17:24.178038 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.177605 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 01:17:24.178038 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.177608 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 01:17:24.178038 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.177610 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 01:17:24.178038 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.177613 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 01:17:24.178038 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.177615 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 01:17:24.178038 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.177618 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 01:17:24.178038 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.177620 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 01:17:24.178038 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.177623 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 01:17:24.178038 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.177626 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 01:17:24.178038 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.177628 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 01:17:24.178517 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.177631 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 01:17:24.178517 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.177634 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 01:17:24.178517 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.177636 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 01:17:24.178517 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.177639 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 01:17:24.178517 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.177642 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 01:17:24.178517 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.177644 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 01:17:24.178517 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.177647 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 01:17:24.178517 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.177651 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 01:17:24.178517 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.177654 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 01:17:24.178517 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.177656 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 01:17:24.178517 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.177659 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 01:17:24.178517 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.177661 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 01:17:24.178517 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.177664 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 01:17:24.178517 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.177667 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 01:17:24.178517 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.177670 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 01:17:24.178517 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.177673 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 01:17:24.178517 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.177675 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 01:17:24.178517 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.177678 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 01:17:24.178517 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.177680 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 01:17:24.178968 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.177683 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 01:17:24.178968 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.177686 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 01:17:24.178968 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.177689 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 01:17:24.178968 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.177692 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 01:17:24.178968 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.177694 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 01:17:24.178968 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.177697 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 01:17:24.178968 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.177699 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 01:17:24.178968 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.177702 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 01:17:24.178968 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.177704 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 01:17:24.178968 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.177707 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 01:17:24.178968 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.177709 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 01:17:24.178968 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.177712 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 01:17:24.178968 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.177714 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 01:17:24.178968 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.177717 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 01:17:24.178968 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.177720 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 01:17:24.178968 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.177723 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 01:17:24.178968 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.177726 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 01:17:24.178968 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.177729 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 01:17:24.178968 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.177732 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 01:17:24.178968 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.177734 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 01:17:24.179461 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.177737 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 01:17:24.179461 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.177740 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 01:17:24.179461 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.177742 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 01:17:24.179461 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.177745 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 01:17:24.179461 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.177747 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 01:17:24.179461 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.177750 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 01:17:24.179461 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.177753 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 01:17:24.179461 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.177756 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 01:17:24.179461 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.177759 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 01:17:24.179461 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.177761 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 01:17:24.179461 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.177764 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 01:17:24.179461 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.177766 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 01:17:24.179461 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.177768 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 01:17:24.179461 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.177771 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 01:17:24.179461 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.177774 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 01:17:24.179461 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.177776 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 01:17:24.179461 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.177779 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 01:17:24.179461 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.177783 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 01:17:24.179461 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.177786 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 01:17:24.179936 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.177789 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 01:17:24.179936 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.177793 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 01:17:24.179936 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.177796 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 01:17:24.179936 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.177798 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 01:17:24.179936 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.177801 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 01:17:24.179936 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.177803 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 01:17:24.179936 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.178586 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 01:17:24.179936 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.178593 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 01:17:24.179936 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.178596 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 01:17:24.179936 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.178600 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 01:17:24.179936 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.178603 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 01:17:24.179936 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.178606 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 01:17:24.179936 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.178609 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 01:17:24.179936 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.178612 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 01:17:24.179936 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.178614 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 01:17:24.179936 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.178617 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 03 01:17:24.179936 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.178620 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 01:17:24.179936 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.178622 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 01:17:24.179936 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.178625 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 01:17:24.179936 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.178628 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 01:17:24.180433 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.178631 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 01:17:24.180433 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.178634 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 01:17:24.180433 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.178636 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 01:17:24.180433 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.178639 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 01:17:24.180433 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.178641 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 01:17:24.180433 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.178644 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 01:17:24.180433 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.178647 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 01:17:24.180433 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.178649 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 01:17:24.180433 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.178652 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 01:17:24.180433 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.178655 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 01:17:24.180433 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.178658 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 01:17:24.180433 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.178660 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 01:17:24.180433 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.178663 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 01:17:24.180433 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.178666 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 01:17:24.180433 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.178668 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 01:17:24.180433 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.178671 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 01:17:24.180433 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.178673 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 01:17:24.180433 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.178676 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 01:17:24.180433 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.178679 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 01:17:24.180433 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.178681 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 01:17:24.180915 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.178683 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 01:17:24.180915 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.178686 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 01:17:24.180915 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.178690 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 01:17:24.180915 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.178693 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 01:17:24.180915 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.178696 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 01:17:24.180915 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.178699 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 01:17:24.180915 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.178701 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 01:17:24.180915 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.178704 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 01:17:24.180915 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.178706 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 01:17:24.180915 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.178709 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 01:17:24.180915 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.178711 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 01:17:24.180915 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.178715 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 01:17:24.180915 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.178717 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 01:17:24.180915 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.178720 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 01:17:24.180915 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.178722 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 01:17:24.180915 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.178725 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 01:17:24.180915 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.178729 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 01:17:24.180915 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.178733 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 01:17:24.180915 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.178736 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 01:17:24.181494 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.178739 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 01:17:24.181494 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.178742 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 01:17:24.181494 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.178745 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 01:17:24.181494 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.178747 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 01:17:24.181494 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.178750 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 01:17:24.181494 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.178752 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 01:17:24.181494 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.178755 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 01:17:24.181494 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.178758 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 01:17:24.181494 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.178760 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 01:17:24.181494 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.178763 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 01:17:24.181494 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.178765 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 01:17:24.181494 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.178768 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 01:17:24.181494 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.178771 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 01:17:24.181494 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.178774 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 01:17:24.181494 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.178777 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 01:17:24.181494 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.178780 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 01:17:24.181494 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.178783 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 01:17:24.181494 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.178786 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 01:17:24.181494 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.178789 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 01:17:24.182214 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.178791 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 01:17:24.182214 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.178794 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 01:17:24.182214 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.178796 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 01:17:24.182214 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.178799 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 01:17:24.182214 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.178802 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 01:17:24.182214 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.178805 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 01:17:24.182214 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.178807 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 01:17:24.182214 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.178810 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 01:17:24.182214 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.178812 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 01:17:24.182214 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.178815 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 01:17:24.182214 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.178818 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 01:17:24.182214 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.178820 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 01:17:24.182214 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.178823 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 01:17:24.182214 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.178832 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 01:17:24.182214 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.178836 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 01:17:24.182214 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.178839 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 01:17:24.182214 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.178841 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 01:17:24.182214 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.178844 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 01:17:24.182214 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.178847 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 01:17:24.182713 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.178849 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 01:17:24.182713 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.178852 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 01:17:24.182713 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.178854 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 01:17:24.182713 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.178857 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 01:17:24.182713 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.178860 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 01:17:24.182713 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.178862 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 01:17:24.182713 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.178865 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 01:17:24.182713 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.178870 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 01:17:24.182713 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.178873 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 01:17:24.182713 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.178876 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 01:17:24.182713 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.178878 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 01:17:24.182713 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.178881 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 01:17:24.182713 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.178886 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 01:17:24.182713 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.178890 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 01:17:24.182713 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.178894 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 01:17:24.182713 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.179393 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 01:17:24.182713 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.179399 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 01:17:24.182713 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.179402 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 01:17:24.182713 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.179406 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 01:17:24.182713 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.179409 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 01:17:24.183287 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.179411 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 01:17:24.183287 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.179414 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 01:17:24.183287 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.179417 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 01:17:24.183287 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.179419 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 01:17:24.183287 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.179422 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 01:17:24.183287 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.179425 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 01:17:24.183287 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.179427 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 01:17:24.183287 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.179431 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 01:17:24.183287 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.179433 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 01:17:24.183287 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.179436 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 01:17:24.183287 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.179439 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 01:17:24.183287 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.179441 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 01:17:24.183287 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.179444 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 01:17:24.183287 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.179447 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 01:17:24.183287 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.179449 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 01:17:24.183287 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.179452 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 01:17:24.183287 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.179454 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 01:17:24.183287 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.179457 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 01:17:24.183287 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.179459 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 01:17:24.183779 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.179463 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 01:17:24.183779 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.179465 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 01:17:24.183779 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.179469 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 01:17:24.183779 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.179472 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 01:17:24.183779 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.179475 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 01:17:24.183779 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.179478 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 01:17:24.183779 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.179480 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 01:17:24.183779 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.179483 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 01:17:24.183779 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.179485 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 01:17:24.183779 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.179488 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 01:17:24.183779 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.179490 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 01:17:24.183779 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.179493 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 01:17:24.183779 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.179495 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 01:17:24.183779 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.179498 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 01:17:24.183779 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.179500 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 01:17:24.183779 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.179503 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 01:17:24.183779 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.179505 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 01:17:24.183779 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.179508 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 01:17:24.183779 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.179511 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 01:17:24.183779 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.179513 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 01:17:24.184257 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.179515 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 01:17:24.184257 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.179518 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 01:17:24.184257 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.179520 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 01:17:24.184257 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.179523 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 01:17:24.184257 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.179525 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 01:17:24.184257 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.179528 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 01:17:24.184257 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.179530 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 01:17:24.184257 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.179533 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 01:17:24.184257 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.179535 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 01:17:24.184257 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.179538 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 01:17:24.184257 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.179540 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 01:17:24.184257 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.179542 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 01:17:24.184257 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.179545 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 01:17:24.184257 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.179548 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 01:17:24.184257 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.179551 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 01:17:24.184257 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.179554 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 01:17:24.184257 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.179556 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 01:17:24.184257 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.179559 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 01:17:24.184257 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.179562 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 01:17:24.184257 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.179564 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 01:17:24.184781 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.179567 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 01:17:24.184781 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.179572 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 01:17:24.184781 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.179575 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 01:17:24.184781 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.179577 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 01:17:24.184781 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.179580 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 01:17:24.184781 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.179582 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 01:17:24.184781 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.179585 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 03 01:17:24.184781 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.179587 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 01:17:24.184781 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.179591 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 01:17:24.184781 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.179595 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 01:17:24.184781 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.179598 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 01:17:24.184781 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.179601 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 01:17:24.184781 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.179604 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 01:17:24.184781 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.179607 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 01:17:24.184781 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.179610 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 01:17:24.184781 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.179612 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 01:17:24.184781 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.179615 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 01:17:24.184781 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.179617 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 01:17:24.184781 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.179620 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 01:17:24.185221 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.179623 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 01:17:24.185221 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.179626 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 01:17:24.185221 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.179628 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 01:17:24.185221 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.179631 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 01:17:24.185221 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.179633 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 01:17:24.185221 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.179636 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 01:17:24.185221 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.179639 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 01:17:24.185221 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.179641 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 01:17:24.185221 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.179644 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 01:17:24.185221 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.179647 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 01:17:24.185221 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.179649 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 01:17:24.185221 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.179652 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 01:17:24.185221 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.179654 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 01:17:24.185221 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.179657 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 01:17:24.185221 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.179661 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 01:17:24.185221 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.179663 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 01:17:24.185221 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.179666 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 01:17:24.185221 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.179670 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 01:17:24.185221 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.179673 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 01:17:24.185676 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.179677 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 01:17:24.185676 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.179680 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 01:17:24.185676 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.179683 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 01:17:24.185676 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.179686 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 01:17:24.185676 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.179776 2578 flags.go:64] FLAG: --address="0.0.0.0" Nov 03 01:17:24.185676 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.179783 2578 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 03 01:17:24.185676 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.179789 2578 flags.go:64] FLAG: --anonymous-auth="true" Nov 03 01:17:24.185676 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.179794 2578 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 03 01:17:24.185676 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.179801 2578 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 03 01:17:24.185676 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.179805 2578 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 03 01:17:24.185676 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.179809 2578 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 03 01:17:24.185676 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.179814 2578 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 03 01:17:24.185676 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.179817 2578 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 03 01:17:24.185676 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.179820 2578 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 03 01:17:24.185676 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.179823 2578 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 03 01:17:24.185676 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.179827 2578 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 03 01:17:24.185676 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.179830 2578 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 03 01:17:24.185676 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.179833 2578 flags.go:64] FLAG: --cgroup-root="" Nov 03 01:17:24.185676 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.179837 2578 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 03 01:17:24.185676 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.179840 2578 flags.go:64] FLAG: --client-ca-file="" Nov 03 01:17:24.185676 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.179843 2578 flags.go:64] FLAG: --cloud-provider="external" Nov 03 01:17:24.185676 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.179846 2578 flags.go:64] FLAG: --cluster-dns="[]" Nov 03 01:17:24.185676 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.179849 2578 flags.go:64] FLAG: --cluster-domain="" Nov 03 01:17:24.186215 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.179852 2578 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 03 01:17:24.186215 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.179855 2578 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 03 01:17:24.186215 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.179859 2578 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 03 01:17:24.186215 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.179863 2578 flags.go:64] FLAG: --container-log-max-files="5" Nov 03 01:17:24.186215 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.179868 2578 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 03 01:17:24.186215 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.179871 2578 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 03 01:17:24.186215 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.179875 2578 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 03 01:17:24.186215 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.179878 2578 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 03 01:17:24.186215 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.179881 2578 flags.go:64] FLAG: --contention-profiling="false" Nov 03 01:17:24.186215 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.179884 2578 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 03 01:17:24.186215 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.179887 2578 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 03 01:17:24.186215 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.179890 2578 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 03 01:17:24.186215 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.179893 2578 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 03 01:17:24.186215 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.179897 2578 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 03 01:17:24.186215 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.179900 2578 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 03 01:17:24.186215 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.179903 2578 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 03 01:17:24.186215 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.179906 2578 flags.go:64] FLAG: --enable-load-reader="false" Nov 03 01:17:24.186215 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.179910 2578 flags.go:64] FLAG: --enable-server="true" Nov 03 01:17:24.186215 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.179914 2578 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 03 01:17:24.186215 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.179918 2578 flags.go:64] FLAG: --event-burst="100" Nov 03 01:17:24.186215 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.179921 2578 flags.go:64] FLAG: --event-qps="50" Nov 03 01:17:24.186215 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.179925 2578 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 03 01:17:24.186215 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.179927 2578 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 03 01:17:24.186215 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.179931 2578 flags.go:64] FLAG: --eviction-hard="" Nov 03 01:17:24.186835 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.179935 2578 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 03 01:17:24.186835 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.179938 2578 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 03 01:17:24.186835 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.179941 2578 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 03 01:17:24.186835 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.179945 2578 flags.go:64] FLAG: --eviction-soft="" Nov 03 01:17:24.186835 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.179948 2578 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 03 01:17:24.186835 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.179951 2578 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 03 01:17:24.186835 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.179954 2578 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 03 01:17:24.186835 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.179957 2578 flags.go:64] FLAG: --experimental-mounter-path="" Nov 03 01:17:24.186835 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.179960 2578 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 03 01:17:24.186835 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.179964 2578 flags.go:64] FLAG: --fail-swap-on="true" Nov 03 01:17:24.186835 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.179967 2578 flags.go:64] FLAG: --feature-gates="" Nov 03 01:17:24.186835 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.179972 2578 flags.go:64] FLAG: --file-check-frequency="20s" Nov 03 01:17:24.186835 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.179976 2578 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 03 01:17:24.186835 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.179980 2578 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 03 01:17:24.186835 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.179983 2578 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 03 01:17:24.186835 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.179987 2578 flags.go:64] FLAG: --healthz-port="10248" Nov 03 01:17:24.186835 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.179990 2578 flags.go:64] FLAG: --help="false" Nov 03 01:17:24.186835 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.179993 2578 flags.go:64] FLAG: --hostname-override="ip-10-0-136-16.us-west-2.compute.internal" Nov 03 01:17:24.186835 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.179996 2578 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 03 01:17:24.186835 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.179999 2578 flags.go:64] FLAG: --http-check-frequency="20s" Nov 03 01:17:24.186835 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.180002 2578 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 03 01:17:24.186835 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.180005 2578 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 03 01:17:24.186835 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.180009 2578 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 03 01:17:24.187414 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.180012 2578 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 03 01:17:24.187414 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.180015 2578 flags.go:64] FLAG: --image-service-endpoint="" Nov 03 01:17:24.187414 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.180019 2578 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 03 01:17:24.187414 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.180022 2578 flags.go:64] FLAG: --kube-api-burst="100" Nov 03 01:17:24.187414 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.180025 2578 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 03 01:17:24.187414 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.180028 2578 flags.go:64] FLAG: --kube-api-qps="50" Nov 03 01:17:24.187414 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.180031 2578 flags.go:64] FLAG: --kube-reserved="" Nov 03 01:17:24.187414 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.180034 2578 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 03 01:17:24.187414 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.180037 2578 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 03 01:17:24.187414 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.180040 2578 flags.go:64] FLAG: --kubelet-cgroups="" Nov 03 01:17:24.187414 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.180043 2578 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 03 01:17:24.187414 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.180046 2578 flags.go:64] FLAG: --lock-file="" Nov 03 01:17:24.187414 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.180048 2578 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 03 01:17:24.187414 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.180051 2578 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 03 01:17:24.187414 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.180055 2578 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 03 01:17:24.187414 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.180061 2578 flags.go:64] FLAG: --log-json-split-stream="false" Nov 03 01:17:24.187414 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.180064 2578 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 03 01:17:24.187414 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.180066 2578 flags.go:64] FLAG: --log-text-split-stream="false" Nov 03 01:17:24.187414 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.180069 2578 flags.go:64] FLAG: --logging-format="text" Nov 03 01:17:24.187414 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.180072 2578 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 03 01:17:24.187414 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.180076 2578 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 03 01:17:24.187414 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.180079 2578 flags.go:64] FLAG: --manifest-url="" Nov 03 01:17:24.187414 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.180082 2578 flags.go:64] FLAG: --manifest-url-header="" Nov 03 01:17:24.187414 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.180087 2578 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 03 01:17:24.187414 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.180090 2578 flags.go:64] FLAG: --max-open-files="1000000" Nov 03 01:17:24.188029 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.180094 2578 flags.go:64] FLAG: --max-pods="110" Nov 03 01:17:24.188029 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.180097 2578 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 03 01:17:24.188029 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.180100 2578 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 03 01:17:24.188029 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.180103 2578 flags.go:64] FLAG: --memory-manager-policy="None" Nov 03 01:17:24.188029 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.180107 2578 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 03 01:17:24.188029 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.180110 2578 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 03 01:17:24.188029 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.180113 2578 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 03 01:17:24.188029 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.180116 2578 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 03 01:17:24.188029 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.180123 2578 flags.go:64] FLAG: --node-status-max-images="50" Nov 03 01:17:24.188029 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.180127 2578 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 03 01:17:24.188029 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.180131 2578 flags.go:64] FLAG: --oom-score-adj="-999" Nov 03 01:17:24.188029 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.180134 2578 flags.go:64] FLAG: --pod-cidr="" Nov 03 01:17:24.188029 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.180137 2578 flags.go:64] FLAG: --pod-manifest-path="" Nov 03 01:17:24.188029 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.180140 2578 flags.go:64] FLAG: --pod-max-pids="-1" Nov 03 01:17:24.188029 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.180143 2578 flags.go:64] FLAG: --pods-per-core="0" Nov 03 01:17:24.188029 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.180146 2578 flags.go:64] FLAG: --port="10250" Nov 03 01:17:24.188029 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.180149 2578 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 03 01:17:24.188029 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.180152 2578 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-0fe7c81c61095ddcc" Nov 03 01:17:24.188029 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.180155 2578 flags.go:64] FLAG: --qos-reserved="" Nov 03 01:17:24.188029 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.180158 2578 flags.go:64] FLAG: --read-only-port="10255" Nov 03 01:17:24.188029 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.180161 2578 flags.go:64] FLAG: --register-node="true" Nov 03 01:17:24.188029 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.180164 2578 flags.go:64] FLAG: --register-with-taints="" Nov 03 01:17:24.188029 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.180167 2578 flags.go:64] FLAG: --registry-burst="10" Nov 03 01:17:24.188029 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.180170 2578 flags.go:64] FLAG: --registry-qps="5" Nov 03 01:17:24.188029 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.180173 2578 flags.go:64] FLAG: --reserved-cpus="" Nov 03 01:17:24.188029 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.180176 2578 flags.go:64] FLAG: --reserved-memory="" Nov 03 01:17:24.188694 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.180180 2578 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 03 01:17:24.188694 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.180183 2578 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 03 01:17:24.188694 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.180187 2578 flags.go:64] FLAG: --rotate-certificates="false" Nov 03 01:17:24.188694 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.180190 2578 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 03 01:17:24.188694 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.180193 2578 flags.go:64] FLAG: --runonce="false" Nov 03 01:17:24.188694 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.180196 2578 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 03 01:17:24.188694 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.180199 2578 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 03 01:17:24.188694 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.180202 2578 flags.go:64] FLAG: --seccomp-default="false" Nov 03 01:17:24.188694 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.180205 2578 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 03 01:17:24.188694 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.180207 2578 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 03 01:17:24.188694 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.180211 2578 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 03 01:17:24.188694 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.180214 2578 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 03 01:17:24.188694 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.180217 2578 flags.go:64] FLAG: --storage-driver-password="root" Nov 03 01:17:24.188694 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.180219 2578 flags.go:64] FLAG: --storage-driver-secure="false" Nov 03 01:17:24.188694 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.180222 2578 flags.go:64] FLAG: --storage-driver-table="stats" Nov 03 01:17:24.188694 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.180227 2578 flags.go:64] FLAG: --storage-driver-user="root" Nov 03 01:17:24.188694 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.180230 2578 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 03 01:17:24.188694 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.180233 2578 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 03 01:17:24.188694 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.180236 2578 flags.go:64] FLAG: --system-cgroups="" Nov 03 01:17:24.188694 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.180239 2578 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 03 01:17:24.188694 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.180245 2578 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 03 01:17:24.188694 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.180248 2578 flags.go:64] FLAG: --tls-cert-file="" Nov 03 01:17:24.188694 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.180251 2578 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 03 01:17:24.188694 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.180254 2578 flags.go:64] FLAG: --tls-min-version="" Nov 03 01:17:24.188694 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.180257 2578 flags.go:64] FLAG: --tls-private-key-file="" Nov 03 01:17:24.189285 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.180260 2578 flags.go:64] FLAG: --topology-manager-policy="none" Nov 03 01:17:24.189285 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.180263 2578 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 03 01:17:24.189285 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.180266 2578 flags.go:64] FLAG: --topology-manager-scope="container" Nov 03 01:17:24.189285 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.180269 2578 flags.go:64] FLAG: --v="2" Nov 03 01:17:24.189285 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.180273 2578 flags.go:64] FLAG: --version="false" Nov 03 01:17:24.189285 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.180277 2578 flags.go:64] FLAG: --vmodule="" Nov 03 01:17:24.189285 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.180282 2578 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 03 01:17:24.189285 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.180285 2578 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 03 01:17:24.189285 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180436 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 01:17:24.189285 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180440 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 01:17:24.189285 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180444 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 01:17:24.189285 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180448 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 01:17:24.189285 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180451 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 01:17:24.189285 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180454 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 01:17:24.189285 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180457 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 01:17:24.189285 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180460 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 01:17:24.189285 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180462 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 01:17:24.189285 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180465 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 01:17:24.189285 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180469 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 01:17:24.189285 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180472 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 01:17:24.189285 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180474 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 01:17:24.189796 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180477 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 01:17:24.189796 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180481 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 01:17:24.189796 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180484 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 01:17:24.189796 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180486 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 01:17:24.189796 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180489 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 01:17:24.189796 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180491 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 01:17:24.189796 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180494 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 01:17:24.189796 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180497 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 01:17:24.189796 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180499 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 01:17:24.189796 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180502 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 01:17:24.189796 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180505 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 01:17:24.189796 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180507 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 01:17:24.189796 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180510 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 01:17:24.189796 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180513 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 01:17:24.189796 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180515 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 01:17:24.189796 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180518 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 01:17:24.189796 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180520 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 01:17:24.189796 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180523 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 01:17:24.189796 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180526 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 01:17:24.189796 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180528 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 01:17:24.190267 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180532 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 01:17:24.190267 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180535 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 01:17:24.190267 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180538 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 01:17:24.190267 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180541 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 01:17:24.190267 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180543 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 01:17:24.190267 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180546 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 01:17:24.190267 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180549 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 01:17:24.190267 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180551 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 01:17:24.190267 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180554 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 01:17:24.190267 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180556 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 01:17:24.190267 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180559 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 01:17:24.190267 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180561 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 01:17:24.190267 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180564 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 01:17:24.190267 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180568 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 01:17:24.190267 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180571 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 01:17:24.190267 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180573 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 01:17:24.190267 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180576 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 01:17:24.190267 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180579 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 01:17:24.190267 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180581 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 01:17:24.190267 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180584 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 01:17:24.190767 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180586 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 01:17:24.190767 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180589 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 01:17:24.190767 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180593 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 01:17:24.190767 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180597 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 01:17:24.190767 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180600 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 01:17:24.190767 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180602 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 01:17:24.190767 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180605 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 01:17:24.190767 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180608 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 01:17:24.190767 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180610 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 01:17:24.190767 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180613 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 01:17:24.190767 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180615 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 01:17:24.190767 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180618 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 01:17:24.190767 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180622 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 01:17:24.190767 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180625 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 01:17:24.190767 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180627 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 01:17:24.190767 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180630 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 01:17:24.190767 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180632 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 01:17:24.190767 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180635 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 01:17:24.190767 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180638 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 01:17:24.191224 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180640 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 01:17:24.191224 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180643 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 01:17:24.191224 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180645 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 01:17:24.191224 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180649 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 01:17:24.191224 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180652 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 03 01:17:24.191224 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180654 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 01:17:24.191224 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180658 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 01:17:24.191224 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180661 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 01:17:24.191224 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180663 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 01:17:24.191224 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180666 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 01:17:24.191224 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180669 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 01:17:24.191224 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180671 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 01:17:24.191224 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180674 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 01:17:24.191224 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180676 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 01:17:24.191224 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180679 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 01:17:24.191224 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180681 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 01:17:24.191224 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180684 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 01:17:24.191224 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180687 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 01:17:24.191224 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180689 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 01:17:24.191696 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180692 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 01:17:24.191696 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180694 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 01:17:24.191696 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180697 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 01:17:24.191696 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180699 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 01:17:24.191696 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180702 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 01:17:24.191696 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180704 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 01:17:24.191696 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180708 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 01:17:24.191696 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180711 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 01:17:24.191696 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180713 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 01:17:24.191696 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180716 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 01:17:24.191696 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180718 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 01:17:24.191696 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180721 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 01:17:24.191696 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180724 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 01:17:24.191696 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180728 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 01:17:24.191696 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180731 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 01:17:24.191696 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.180756 2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 01:17:24.192082 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180793 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 01:17:24.192082 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180798 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 01:17:24.192082 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180801 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 01:17:24.192082 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180803 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 01:17:24.192082 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180807 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 01:17:24.192082 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180810 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 01:17:24.192082 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180813 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 01:17:24.192082 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180815 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 01:17:24.192082 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180818 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 01:17:24.192082 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180820 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 01:17:24.192082 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180823 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 01:17:24.192082 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180826 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 01:17:24.192082 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180828 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 01:17:24.192082 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180831 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 01:17:24.192082 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180833 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 01:17:24.192082 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180836 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 01:17:24.192082 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180838 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 01:17:24.192082 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180841 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 01:17:24.192082 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180843 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 01:17:24.192082 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180846 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 01:17:24.192572 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180848 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 01:17:24.192572 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180851 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 01:17:24.192572 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180855 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 01:17:24.192572 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180858 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 01:17:24.192572 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180860 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 01:17:24.192572 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180863 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 01:17:24.192572 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180866 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 01:17:24.192572 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180868 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 01:17:24.192572 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180871 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 01:17:24.192572 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180874 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 01:17:24.192572 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180876 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 01:17:24.192572 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180879 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 01:17:24.192572 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180881 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 01:17:24.192572 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180884 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 01:17:24.192572 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180887 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 01:17:24.192572 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180889 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 01:17:24.192572 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180893 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 01:17:24.192572 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180896 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 01:17:24.192572 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180898 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 01:17:24.192572 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180901 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 01:17:24.193049 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180903 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 01:17:24.193049 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180906 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 01:17:24.193049 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180908 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 01:17:24.193049 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180911 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 01:17:24.193049 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180913 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 01:17:24.193049 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180916 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 01:17:24.193049 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180919 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 01:17:24.193049 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180922 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 01:17:24.193049 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180924 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 01:17:24.193049 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180927 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 01:17:24.193049 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180930 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 01:17:24.193049 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180932 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 01:17:24.193049 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180935 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 01:17:24.193049 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180938 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 01:17:24.193049 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180941 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 01:17:24.193049 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180945 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 01:17:24.193049 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180949 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 01:17:24.193049 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180953 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 01:17:24.193049 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180955 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 01:17:24.193049 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180958 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 01:17:24.193551 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180960 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 01:17:24.193551 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180963 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 01:17:24.193551 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180966 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 01:17:24.193551 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180969 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 01:17:24.193551 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180971 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 01:17:24.193551 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180974 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 01:17:24.193551 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180976 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 01:17:24.193551 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180978 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 01:17:24.193551 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180983 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 01:17:24.193551 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180986 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 01:17:24.193551 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180989 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 01:17:24.193551 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180992 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 01:17:24.193551 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180995 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 01:17:24.193551 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.180998 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 01:17:24.193551 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.181000 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 01:17:24.193551 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.181003 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 01:17:24.193551 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.181006 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 03 01:17:24.193551 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.181008 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 01:17:24.193551 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.181011 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 01:17:24.194056 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.181013 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 01:17:24.194056 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.181016 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 01:17:24.194056 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.181019 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 01:17:24.194056 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.181021 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 01:17:24.194056 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.181024 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 01:17:24.194056 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.181026 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 01:17:24.194056 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.181029 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 01:17:24.194056 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.181032 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 01:17:24.194056 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.181035 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 01:17:24.194056 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.181037 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 01:17:24.194056 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.181040 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 01:17:24.194056 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.181042 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 01:17:24.194056 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.181045 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 01:17:24.194056 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.181047 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 01:17:24.194056 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.181050 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 01:17:24.194056 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.181053 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 01:17:24.194056 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.181056 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 01:17:24.194056 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.181058 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 01:17:24.194056 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.181061 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 01:17:24.194527 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.181063 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 01:17:24.194527 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.181066 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 01:17:24.194527 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.181069 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 01:17:24.194527 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.181072 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 01:17:24.194527 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.181074 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 01:17:24.194527 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.181077 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 01:17:24.194527 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.181081 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 01:17:24.194527 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.181085 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 01:17:24.194527 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.189643 2578 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 03 01:17:24.194527 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.189659 2578 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 03 01:17:24.194527 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.189676 2578 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 03 01:17:24.194527 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.189680 2578 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 03 01:17:24.194527 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.189739 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 01:17:24.194527 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.189745 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 01:17:24.194527 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.189749 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 01:17:24.194527 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.189752 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 01:17:24.194527 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.189755 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 01:17:24.194527 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.189757 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 01:17:24.194527 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.189761 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 01:17:24.194998 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.189763 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 03 01:17:24.194998 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.189767 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 01:17:24.194998 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.189770 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 01:17:24.194998 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.189772 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 01:17:24.194998 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.189775 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 01:17:24.194998 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.189777 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 01:17:24.194998 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.189780 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 01:17:24.194998 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.189783 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 01:17:24.194998 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.189786 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 01:17:24.194998 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.189789 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 01:17:24.194998 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.189791 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 01:17:24.194998 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.189794 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 01:17:24.194998 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.189797 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 01:17:24.194998 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.189800 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 01:17:24.194998 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.189803 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 01:17:24.194998 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.189807 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 01:17:24.194998 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.189810 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 01:17:24.194998 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.189813 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 01:17:24.194998 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.189816 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 01:17:24.195462 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.189818 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 01:17:24.195462 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.189821 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 01:17:24.195462 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.189823 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 01:17:24.195462 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.189826 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 01:17:24.195462 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.189828 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 01:17:24.195462 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.189831 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 01:17:24.195462 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.189834 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 01:17:24.195462 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.189837 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 01:17:24.195462 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.189839 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 01:17:24.195462 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.189842 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 01:17:24.195462 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.189844 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 01:17:24.195462 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.189847 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 01:17:24.195462 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.189849 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 01:17:24.195462 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.189852 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 01:17:24.195462 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.189855 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 01:17:24.195462 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.189857 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 01:17:24.195462 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.189860 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 01:17:24.195462 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.189863 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 01:17:24.195462 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.189866 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 01:17:24.195901 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.189868 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 01:17:24.195901 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.189871 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 01:17:24.195901 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.189873 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 01:17:24.195901 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.189876 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 01:17:24.195901 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.189879 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 01:17:24.195901 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.189881 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 01:17:24.195901 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.189884 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 01:17:24.195901 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.189887 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 01:17:24.195901 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.189889 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 01:17:24.195901 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.189893 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 01:17:24.195901 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.189896 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 01:17:24.195901 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.189898 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 01:17:24.195901 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.189901 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 01:17:24.195901 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.189903 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 01:17:24.195901 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.189906 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 01:17:24.195901 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.189908 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 01:17:24.195901 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.189911 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 01:17:24.195901 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.189914 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 01:17:24.195901 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.189916 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 01:17:24.195901 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.189919 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 01:17:24.196400 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.189922 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 01:17:24.196400 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.189926 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 01:17:24.196400 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.189929 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 01:17:24.196400 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.189932 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 01:17:24.196400 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.189935 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 01:17:24.196400 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.189938 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 01:17:24.196400 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.189941 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 01:17:24.196400 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.189943 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 01:17:24.196400 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.189946 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 01:17:24.196400 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.189949 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 01:17:24.196400 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.189951 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 01:17:24.196400 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.189954 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 01:17:24.196400 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.189956 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 01:17:24.196400 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.189959 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 01:17:24.196400 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.189962 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 01:17:24.196400 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.189966 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 01:17:24.196400 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.189971 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 01:17:24.196400 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.189974 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 01:17:24.196400 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.189977 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 01:17:24.196400 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.189980 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 01:17:24.196877 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.189982 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 01:17:24.196877 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.189986 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 01:17:24.196877 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.189989 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 01:17:24.196877 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.189992 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 01:17:24.196877 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.189995 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 01:17:24.196877 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.189997 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 01:17:24.196877 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190000 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 01:17:24.196877 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190003 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 01:17:24.196877 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190006 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 01:17:24.196877 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190008 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 01:17:24.196877 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190011 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 01:17:24.196877 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190014 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 01:17:24.196877 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190016 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 01:17:24.196877 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190019 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 01:17:24.196877 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190022 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 01:17:24.196877 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190024 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 01:17:24.196877 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190027 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 01:17:24.196877 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190029 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 01:17:24.196877 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190032 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 01:17:24.197313 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190035 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 01:17:24.197313 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190037 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 01:17:24.197313 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.190067 2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 01:17:24.197313 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190212 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 01:17:24.197313 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190217 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 01:17:24.197313 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190220 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 01:17:24.197313 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190223 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 01:17:24.197313 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190228 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 01:17:24.197313 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190232 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 01:17:24.197313 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190236 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 01:17:24.197313 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190239 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 01:17:24.197313 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190242 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 01:17:24.197313 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190245 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 01:17:24.197313 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190248 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 01:17:24.197313 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190250 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 01:17:24.197698 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190253 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 01:17:24.197698 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190256 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 01:17:24.197698 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190259 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 01:17:24.197698 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190261 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 01:17:24.197698 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190264 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 01:17:24.197698 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190266 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 01:17:24.197698 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190269 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 01:17:24.197698 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190272 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 01:17:24.197698 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190275 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 01:17:24.197698 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190277 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 01:17:24.197698 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190280 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 01:17:24.197698 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190283 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 01:17:24.197698 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190286 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 01:17:24.197698 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190288 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 01:17:24.197698 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190291 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 01:17:24.197698 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190293 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 01:17:24.197698 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190297 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 01:17:24.197698 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190299 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 01:17:24.197698 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190302 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 01:17:24.198148 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190304 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 01:17:24.198148 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190307 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 01:17:24.198148 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190310 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 01:17:24.198148 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190313 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 01:17:24.198148 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190315 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 01:17:24.198148 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190318 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 01:17:24.198148 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190321 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 01:17:24.198148 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190324 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 01:17:24.198148 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190326 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 01:17:24.198148 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190329 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 01:17:24.198148 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190331 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 01:17:24.198148 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190334 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 01:17:24.198148 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190337 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 01:17:24.198148 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190340 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 01:17:24.198148 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190342 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 01:17:24.198148 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190345 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 01:17:24.198148 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190348 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 01:17:24.198148 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190350 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 01:17:24.198148 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190353 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 01:17:24.198148 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190355 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 01:17:24.198645 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190358 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 01:17:24.198645 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190375 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 01:17:24.198645 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190380 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 01:17:24.198645 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190383 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 01:17:24.198645 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190385 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 01:17:24.198645 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190388 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 01:17:24.198645 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190391 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 01:17:24.198645 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190394 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 01:17:24.198645 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190396 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 01:17:24.198645 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190399 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 01:17:24.198645 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190402 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 01:17:24.198645 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190404 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 01:17:24.198645 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190407 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 01:17:24.198645 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190409 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 01:17:24.198645 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190412 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 01:17:24.198645 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190414 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 01:17:24.198645 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190417 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 01:17:24.198645 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190419 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 01:17:24.198645 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190422 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 01:17:24.199123 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190427 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 01:17:24.199123 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190430 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 01:17:24.199123 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190433 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 01:17:24.199123 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190436 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 01:17:24.199123 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190439 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 01:17:24.199123 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190441 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 01:17:24.199123 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190444 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 01:17:24.199123 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190446 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 01:17:24.199123 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190449 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 01:17:24.199123 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190451 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 01:17:24.199123 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190454 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 01:17:24.199123 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190456 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 01:17:24.199123 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190459 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 01:17:24.199123 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190462 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 01:17:24.199123 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190464 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 01:17:24.199123 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190466 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 01:17:24.199123 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190469 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 01:17:24.199123 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190472 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 01:17:24.199123 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190474 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 01:17:24.199573 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190477 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 01:17:24.199573 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190479 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 01:17:24.199573 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190482 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 01:17:24.199573 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190484 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 01:17:24.199573 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190487 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 01:17:24.199573 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190490 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 01:17:24.199573 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190492 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 01:17:24.199573 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190495 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 01:17:24.199573 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190500 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 01:17:24.199573 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190502 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 03 01:17:24.199573 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190505 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 01:17:24.199573 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190508 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 01:17:24.199573 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190510 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 01:17:24.199573 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190513 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 01:17:24.199573 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190516 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 01:17:24.199573 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190519 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 01:17:24.199573 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190522 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 01:17:24.200047 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.190546 2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 01:17:24.200047 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190598 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 01:17:24.200047 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190604 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 01:17:24.200047 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190607 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 01:17:24.200047 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190610 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 01:17:24.200047 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190613 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 03 01:17:24.200047 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190615 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 01:17:24.200047 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190618 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 01:17:24.200047 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190621 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 01:17:24.200047 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190623 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 01:17:24.200047 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190626 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 01:17:24.200047 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190629 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 01:17:24.200047 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190632 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 01:17:24.200047 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190634 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 01:17:24.200047 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190637 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 01:17:24.200047 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190640 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 01:17:24.200527 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190642 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 01:17:24.200527 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190646 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 01:17:24.200527 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190651 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 01:17:24.200527 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190654 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 01:17:24.200527 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190657 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 01:17:24.200527 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190659 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 01:17:24.200527 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190662 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 01:17:24.200527 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190664 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 01:17:24.200527 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190668 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 01:17:24.200527 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190671 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 01:17:24.200527 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190674 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 01:17:24.200527 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190676 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 01:17:24.200527 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190679 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 01:17:24.200527 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190682 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 01:17:24.200527 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190689 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 01:17:24.200527 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190692 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 01:17:24.200527 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190694 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 01:17:24.200527 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190697 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 01:17:24.201001 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190700 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 01:17:24.201001 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190702 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 01:17:24.201001 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190705 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 01:17:24.201001 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190708 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 01:17:24.201001 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190710 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 01:17:24.201001 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190713 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 01:17:24.201001 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190715 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 01:17:24.201001 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190718 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 01:17:24.201001 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190720 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 01:17:24.201001 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190723 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 01:17:24.201001 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190725 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 01:17:24.201001 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190728 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 01:17:24.201001 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190731 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 01:17:24.201001 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190734 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 01:17:24.201001 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190737 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 01:17:24.201001 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190739 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 01:17:24.201001 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190742 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 01:17:24.201001 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190744 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 01:17:24.201001 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190747 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 01:17:24.201001 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190750 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 01:17:24.201491 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190752 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 01:17:24.201491 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190755 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 01:17:24.201491 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190759 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 01:17:24.201491 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190761 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 01:17:24.201491 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190764 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 01:17:24.201491 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190767 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 01:17:24.201491 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190770 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 01:17:24.201491 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190772 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 01:17:24.201491 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190775 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 01:17:24.201491 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190779 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 01:17:24.201491 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190782 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 01:17:24.201491 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190785 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 01:17:24.201491 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190788 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 01:17:24.201491 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190790 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 01:17:24.201491 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190792 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 01:17:24.201491 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190796 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 01:17:24.201491 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190798 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 01:17:24.201491 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190800 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 01:17:24.201491 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190803 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 01:17:24.201491 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190806 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 01:17:24.202137 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190808 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 01:17:24.202137 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190811 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 01:17:24.202137 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190814 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 01:17:24.202137 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190816 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 01:17:24.202137 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190819 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 01:17:24.202137 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190821 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 01:17:24.202137 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190824 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 01:17:24.202137 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190836 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 01:17:24.202137 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190839 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 01:17:24.202137 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190843 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 01:17:24.202137 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190848 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 01:17:24.202137 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190851 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 01:17:24.202137 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190854 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 01:17:24.202137 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190857 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 01:17:24.202137 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190861 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 01:17:24.202137 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190864 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 01:17:24.202137 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190866 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 01:17:24.202137 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190869 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 01:17:24.202137 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190872 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 01:17:24.202785 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190875 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 01:17:24.202785 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190877 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 01:17:24.202785 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190880 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 01:17:24.202785 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190883 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 01:17:24.202785 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190886 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 01:17:24.202785 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190888 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 01:17:24.202785 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190891 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 01:17:24.202785 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190893 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 01:17:24.202785 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190896 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 01:17:24.202785 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190898 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 01:17:24.202785 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190901 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 01:17:24.202785 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190903 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 01:17:24.202785 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190906 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 01:17:24.202785 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:24.190908 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 01:17:24.202785 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.190961 2578 server.go:957] "Client rotation is on, will bootstrap in background" Nov 03 01:17:24.202785 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.193565 2578 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 03 01:17:24.202785 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.194510 2578 server.go:1014] "Starting client certificate rotation" Nov 03 01:17:24.202785 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.194607 2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 01:17:24.203215 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.195314 2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 01:17:24.203215 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.196249 2578 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 01:17:24.203215 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.197388 2578 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 01:17:24.206924 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.206906 2578 log.go:25] "Validated CRI v1 runtime API" Nov 03 01:17:24.210628 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.210615 2578 log.go:25] "Validated CRI v1 image API" Nov 03 01:17:24.211461 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.211431 2578 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 03 01:17:24.214070 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.214052 2578 fs.go:135] Filesystem UUIDs: map[3cb574a7-0402-4d32-a20a-db0a7770476a:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 8a1ba601-aa3f-4674-8909-74086aba29ec:/dev/nvme0n1p4] Nov 03 01:17:24.214165 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.214070 2578 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 03 01:17:24.220889 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.220758 2578 manager.go:217] Machine: {Timestamp:2026-11-03 01:17:24.219478263 +0000 UTC m=+0.306333835 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100651 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2737fedf27e71eb9003a4a4870e7a4 SystemUUID:ec2737fe-df27-e71e-b900-3a4a4870e7a4 BootID:fbdbed05-f2cf-42d7-91da-8abea231b5a2 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:06:ff:ca:24:12:bb Speed:0 Mtu:9001} {Name:ens5 MacAddress:06:ff:ca:24:12:bb Speed:0 Mtu:9001} {Name:ovs-system MacAddress:f6:a6:c1:38:f3:6a Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 03 01:17:24.221297 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.221285 2578 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 03 01:17:24.221398 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.221386 2578 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 03 01:17:24.224160 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.224132 2578 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 03 01:17:24.224335 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.224163 2578 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-16.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 03 01:17:24.224398 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.224338 2578 topology_manager.go:143] "Creating topology manager with none policy" Nov 03 01:17:24.224398 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.224347 2578 container_manager_linux.go:311] "Creating device plugin manager" Nov 03 01:17:24.224450 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.224398 2578 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 01:17:24.224450 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.224418 2578 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 01:17:24.225210 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.225200 2578 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 03 01:17:24.228052 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.228039 2578 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 03 01:17:24.228179 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.228169 2578 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 03 01:17:24.228837 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.228828 2578 kubelet.go:493] "Attempting to sync node with API server" Nov 03 01:17:24.228868 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.228841 2578 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 03 01:17:24.228868 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.228852 2578 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 03 01:17:24.228868 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.228861 2578 kubelet.go:405] "Adding apiserver pod source" Nov 03 01:17:24.228956 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.228870 2578 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 03 01:17:24.229189 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.229178 2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 01:17:24.231094 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.231077 2578 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 03 01:17:24.234611 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.234596 2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 01:17:24.235468 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.235454 2578 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 03 01:17:24.235542 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.235495 2578 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 03 01:17:24.236057 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.236041 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 03 01:17:24.236116 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.236071 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 03 01:17:24.236116 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.236088 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 03 01:17:24.236116 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.236099 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 03 01:17:24.236116 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.236115 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 03 01:17:24.236234 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.236141 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 03 01:17:24.236472 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.236462 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 03 01:17:24.236527 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.236475 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 03 01:17:24.236527 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.236486 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 03 01:17:24.236527 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.236495 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 03 01:17:24.236527 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.236517 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 03 01:17:24.236701 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.236532 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 03 01:17:24.237162 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.237152 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 03 01:17:24.237214 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.237166 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 03 01:17:24.240888 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.240873 2578 server.go:1263] "Started kubelet" Nov 03 01:17:24.240996 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.240918 2578 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 03 01:17:24.241103 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.241011 2578 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 03 01:17:24.241143 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.241130 2578 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 03 01:17:24.241628 ip-10-0-136-16 systemd[1]: Started Kubernetes Kubelet. Nov 03 01:17:24.242601 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.242176 2578 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 03 01:17:24.244231 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.244216 2578 server.go:317] "Adding debug handlers to kubelet server" Nov 03 01:17:24.245673 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:17:24.245648 2578 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 03 01:17:24.248088 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.248057 2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 03 01:17:24.249039 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.249023 2578 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 03 01:17:24.249136 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:17:24.249114 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-16.us-west-2.compute.internal\" not found" Nov 03 01:17:24.249939 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.249645 2578 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 03 01:17:24.249939 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.249665 2578 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 03 01:17:24.249939 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.249667 2578 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 03 01:17:24.249939 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.249818 2578 reconstruct.go:97] "Volume reconstruction finished" Nov 03 01:17:24.249939 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.249839 2578 reconciler.go:29] "Reconciler: start to sync state" Nov 03 01:17:24.250756 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.250742 2578 factory.go:55] Registering systemd factory Nov 03 01:17:24.250756 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.250756 2578 factory.go:223] Registration of the systemd container factory successfully Nov 03 01:17:24.250974 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.250962 2578 factory.go:153] Registering CRI-O factory Nov 03 01:17:24.251022 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.250976 2578 factory.go:223] Registration of the crio container factory successfully Nov 03 01:17:24.251073 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.251068 2578 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 03 01:17:24.251127 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.251083 2578 factory.go:103] Registering Raw factory Nov 03 01:17:24.251127 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.251096 2578 manager.go:1199] Started watching for new ooms in manager Nov 03 01:17:24.251969 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.251957 2578 manager.go:319] Starting recovery of all containers Nov 03 01:17:24.264764 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.264745 2578 manager.go:324] Recovery completed Nov 03 01:17:24.269048 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.269031 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 01:17:24.271355 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.271337 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-16.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 01:17:24.271448 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.271391 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-16.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 01:17:24.271448 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.271410 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-16.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 01:17:24.271956 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.271942 2578 cpu_manager.go:226] "Starting" policy="none" Nov 03 01:17:24.271956 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.271953 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 01:17:24.272052 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.271954 2578 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 03 01:17:24.272052 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.272021 2578 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 03 01:17:24.272722 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.272705 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-16.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 01:17:24.272780 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.272737 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-16.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 01:17:24.272780 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.272762 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-16.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 01:17:24.350131 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:17:24.350105 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-16.us-west-2.compute.internal\" not found" Nov 03 01:17:24.378920 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.378867 2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-136-16.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 03 01:17:24.410032 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.410010 2578 policy_none.go:50] "Start" Nov 03 01:17:24.410032 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.410032 2578 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 03 01:17:24.410156 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.410049 2578 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 03 01:17:24.411186 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.411174 2578 policy_none.go:44] "Start" Nov 03 01:17:24.424106 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:17:24.424054 2578 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-136-16.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Nov 03 01:17:24.425065 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:17:24.424002 2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-16.us-west-2.compute.internal.18e46420f0bb0b7d default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-16.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-136-16.us-west-2.compute.internal,},FirstTimestamp:2026-11-03 01:17:24.240821117 +0000 UTC m=+0.327676688,LastTimestamp:2026-11-03 01:17:24.240821117 +0000 UTC m=+0.327676688,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-16.us-west-2.compute.internal,}" Nov 03 01:17:24.443946 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.443915 2578 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 03 01:17:24.446524 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.446509 2578 manager.go:341] "Starting Device Plugin manager" Nov 03 01:17:24.446620 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:17:24.446546 2578 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 03 01:17:24.446620 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.446558 2578 server.go:87] "Starting device plugin registration server" Nov 03 01:17:24.446906 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.446892 2578 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 03 01:17:24.446964 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.446908 2578 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 03 01:17:24.447032 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.447010 2578 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 03 01:17:24.447562 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.447100 2578 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 03 01:17:24.447562 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.447114 2578 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 03 01:17:24.448597 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:17:24.448578 2578 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 03 01:17:24.448677 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:17:24.448629 2578 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-16.us-west-2.compute.internal\" not found" Nov 03 01:17:24.508670 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.508488 2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 03 01:17:24.509911 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.509880 2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 03 01:17:24.510013 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.509915 2578 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 03 01:17:24.510013 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.509945 2578 kubelet.go:2521] "Starting kubelet main sync loop" Nov 03 01:17:24.510013 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:17:24.509990 2578 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Nov 03 01:17:24.519910 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.519888 2578 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 03 01:17:24.520023 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.519949 2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-wz6d9" Nov 03 01:17:24.538542 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:17:24.538514 2578 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-136-16.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Nov 03 01:17:24.538542 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:17:24.538513 2578 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Nov 03 01:17:24.538691 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:17:24.538559 2578 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Nov 03 01:17:24.538691 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:17:24.538464 2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-16.us-west-2.compute.internal.18e46420f28cf3ce default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-16.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-136-16.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-136-16.us-west-2.compute.internal,},FirstTimestamp:2026-11-03 01:17:24.27135483 +0000 UTC m=+0.358210442,LastTimestamp:2026-11-03 01:17:24.27135483 +0000 UTC m=+0.358210442,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-16.us-west-2.compute.internal,}" Nov 03 01:17:24.547110 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.547092 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 01:17:24.547991 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.547972 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-16.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 01:17:24.548077 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.548011 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-16.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 01:17:24.548077 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.548034 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-16.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 01:17:24.548077 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.548060 2578 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-136-16.us-west-2.compute.internal" Nov 03 01:17:24.610920 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.610889 2578 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-16.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-16.us-west-2.compute.internal"] Nov 03 01:17:24.611048 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.610967 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 01:17:24.612161 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.612143 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-16.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 01:17:24.612264 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.612180 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-16.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 01:17:24.612264 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.612196 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-16.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 01:17:24.613659 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.613643 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 01:17:24.613801 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.613785 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-16.us-west-2.compute.internal" Nov 03 01:17:24.613851 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.613819 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 01:17:24.614456 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.614436 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-16.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 01:17:24.614547 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.614465 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-16.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 01:17:24.614547 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.614476 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-16.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 01:17:24.614547 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.614437 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-16.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 01:17:24.614547 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.614535 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-16.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 01:17:24.614736 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.614564 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-16.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 01:17:24.615699 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.615682 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-16.us-west-2.compute.internal" Nov 03 01:17:24.615782 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.615714 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 01:17:24.616449 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.616434 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-16.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 01:17:24.616501 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.616465 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-16.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 01:17:24.616501 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.616478 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-16.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 01:17:24.640926 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:17:24.640902 2578 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-16.us-west-2.compute.internal\" not found" node="ip-10-0-136-16.us-west-2.compute.internal" Nov 03 01:17:24.644426 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:17:24.644409 2578 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-16.us-west-2.compute.internal\" not found" node="ip-10-0-136-16.us-west-2.compute.internal" Nov 03 01:17:24.651829 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.651812 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f7e070a0c923c611d3cf96ab59dc8b57-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-16.us-west-2.compute.internal\" (UID: \"f7e070a0c923c611d3cf96ab59dc8b57\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-16.us-west-2.compute.internal" Nov 03 01:17:24.651911 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.651841 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f7e070a0c923c611d3cf96ab59dc8b57-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-16.us-west-2.compute.internal\" (UID: \"f7e070a0c923c611d3cf96ab59dc8b57\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-16.us-west-2.compute.internal" Nov 03 01:17:24.651911 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.651860 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7f364a286e3c240d587086e0ec2180aa-config\") pod \"kube-apiserver-proxy-ip-10-0-136-16.us-west-2.compute.internal\" (UID: \"7f364a286e3c240d587086e0ec2180aa\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-16.us-west-2.compute.internal" Nov 03 01:17:24.703873 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.703818 2578 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 01:17:24.752858 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.752828 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f7e070a0c923c611d3cf96ab59dc8b57-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-16.us-west-2.compute.internal\" (UID: \"f7e070a0c923c611d3cf96ab59dc8b57\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-16.us-west-2.compute.internal" Nov 03 01:17:24.752858 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.752860 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f7e070a0c923c611d3cf96ab59dc8b57-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-16.us-west-2.compute.internal\" (UID: \"f7e070a0c923c611d3cf96ab59dc8b57\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-16.us-west-2.compute.internal" Nov 03 01:17:24.753025 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.752880 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7f364a286e3c240d587086e0ec2180aa-config\") pod \"kube-apiserver-proxy-ip-10-0-136-16.us-west-2.compute.internal\" (UID: \"7f364a286e3c240d587086e0ec2180aa\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-16.us-west-2.compute.internal" Nov 03 01:17:24.753025 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.752920 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f7e070a0c923c611d3cf96ab59dc8b57-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-16.us-west-2.compute.internal\" (UID: \"f7e070a0c923c611d3cf96ab59dc8b57\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-16.us-west-2.compute.internal" Nov 03 01:17:24.753025 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.752940 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f7e070a0c923c611d3cf96ab59dc8b57-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-16.us-west-2.compute.internal\" (UID: \"f7e070a0c923c611d3cf96ab59dc8b57\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-16.us-west-2.compute.internal" Nov 03 01:17:24.753025 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.752948 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7f364a286e3c240d587086e0ec2180aa-config\") pod \"kube-apiserver-proxy-ip-10-0-136-16.us-west-2.compute.internal\" (UID: \"7f364a286e3c240d587086e0ec2180aa\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-16.us-west-2.compute.internal" Nov 03 01:17:24.754496 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.754483 2578 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-136-16.us-west-2.compute.internal" Nov 03 01:17:24.754538 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:17:24.754502 2578 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-136-16.us-west-2.compute.internal\": node \"ip-10-0-136-16.us-west-2.compute.internal\" not found" Nov 03 01:17:24.876858 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:17:24.876826 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-16.us-west-2.compute.internal\" not found" Nov 03 01:17:24.943018 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.942993 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-16.us-west-2.compute.internal" Nov 03 01:17:24.946469 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:24.946449 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-16.us-west-2.compute.internal" Nov 03 01:17:24.976964 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:17:24.976897 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-16.us-west-2.compute.internal\" not found" Nov 03 01:17:25.077637 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:17:25.077588 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-16.us-west-2.compute.internal\" not found" Nov 03 01:17:25.177814 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:17:25.177777 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-16.us-west-2.compute.internal\" not found" Nov 03 01:17:25.195267 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:25.195238 2578 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 03 01:17:25.195460 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:25.195435 2578 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 03 01:17:25.195532 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:17:25.195441 2578 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://aece31dc849e6402abdece8830c7a683-e1048f7a7a1c1f18.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-136-16.us-west-2.compute.internal.18e46420f28da08d\": read tcp 10.0.136.16:50800->3.150.166.25:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-136-16.us-west-2.compute.internal.18e46420f28da08d default 5308 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-16.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-136-16.us-west-2.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-136-16.us-west-2.compute.internal,},FirstTimestamp:2026-11-03 01:17:24 +0000 UTC,LastTimestamp:2026-11-03 01:17:24.548019285 +0000 UTC m=+0.634874859,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-16.us-west-2.compute.internal,}" Nov 03 01:17:25.279226 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:17:25.279141 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-16.us-west-2.compute.internal\" not found" Nov 03 01:17:25.380348 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:17:25.380310 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-16.us-west-2.compute.internal\" not found" Nov 03 01:17:25.418831 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:25.418805 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 01:17:25.419908 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:25.419889 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-16.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 01:17:25.420003 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:25.419922 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-16.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 01:17:25.420003 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:25.419935 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-16.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 01:17:25.480493 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:17:25.480462 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-16.us-west-2.compute.internal\" not found" Nov 03 01:17:25.523835 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:25.523802 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 01:12:25 +0000 UTC" deadline="2028-06-13 12:08:21.806787239 +0000 UTC" Nov 03 01:17:25.523835 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:25.523828 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14122h50m56.282961932s" Nov 03 01:17:25.581486 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:17:25.581425 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-16.us-west-2.compute.internal\" not found" Nov 03 01:17:25.675856 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:25.675820 2578 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 01:17:25.682521 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:17:25.682495 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-16.us-west-2.compute.internal\" not found" Nov 03 01:17:25.783591 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:17:25.783560 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-16.us-west-2.compute.internal\" not found" Nov 03 01:17:25.884934 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:17:25.884848 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-16.us-west-2.compute.internal\" not found" Nov 03 01:17:25.896936 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:25.896913 2578 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 01:17:25.986065 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:17:25.986040 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-16.us-west-2.compute.internal\" not found" Nov 03 01:17:26.087062 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:17:26.087032 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-16.us-west-2.compute.internal\" not found" Nov 03 01:17:26.104081 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.104055 2578 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 01:17:26.126277 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:26.126245 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f364a286e3c240d587086e0ec2180aa.slice/crio-640208824d01bacb81bae87037315951d0d6301e18043589626cbb13a7a78d2b WatchSource:0}: Error finding container 640208824d01bacb81bae87037315951d0d6301e18043589626cbb13a7a78d2b: Status 404 returned error can't find the container with id 640208824d01bacb81bae87037315951d0d6301e18043589626cbb13a7a78d2b Nov 03 01:17:26.126592 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:26.126574 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7e070a0c923c611d3cf96ab59dc8b57.slice/crio-a57cfab1072e0e4b06e4b4217747174d1f2639b037b9403811af934a57dc23e5 WatchSource:0}: Error finding container a57cfab1072e0e4b06e4b4217747174d1f2639b037b9403811af934a57dc23e5: Status 404 returned error can't find the container with id a57cfab1072e0e4b06e4b4217747174d1f2639b037b9403811af934a57dc23e5 Nov 03 01:17:26.132247 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.132231 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 01:17:26.150357 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.150329 2578 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-16.us-west-2.compute.internal" Nov 03 01:17:26.218142 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.218114 2578 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 01:17:26.219180 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.219165 2578 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-16.us-west-2.compute.internal" Nov 03 01:17:26.231272 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.231252 2578 apiserver.go:52] "Watching apiserver" Nov 03 01:17:26.249771 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.249750 2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 03 01:17:26.288546 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.288514 2578 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 01:17:26.346213 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.346181 2578 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 03 01:17:26.346551 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.346527 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-136-16.us-west-2.compute.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2jrfl","openshift-cluster-node-tuning-operator/tuned-c6r2x","openshift-network-diagnostics/network-check-target-kg99x","kube-system/konnectivity-agent-dd9jt","openshift-multus/multus-additional-cni-plugins-jnvg6","openshift-dns/node-resolver-5qr9g","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-16.us-west-2.compute.internal","openshift-multus/network-metrics-daemon-8g56n","openshift-image-registry/node-ca-jpcqz","openshift-multus/multus-44p7d","openshift-network-operator/iptables-alerter-x4tvh","openshift-ovn-kubernetes/ovnkube-node-n2cjg","kube-system/global-pull-secret-syncer-dfr9t"] Nov 03 01:17:26.349063 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.349044 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5qr9g" Nov 03 01:17:26.350265 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.350247 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-n2cjg" Nov 03 01:17:26.351552 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.351530 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-c6r2x" Nov 03 01:17:26.352851 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.352831 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kg99x" Nov 03 01:17:26.352926 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:17:26.352896 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kg99x" podUID="3b2bf2f6-28aa-4480-9f4a-f4e7f659e966" Nov 03 01:17:26.354535 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.354511 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-dd9jt" Nov 03 01:17:26.356331 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.356318 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-jnvg6" Nov 03 01:17:26.357754 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.357734 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8g56n" Nov 03 01:17:26.357852 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:17:26.357808 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8g56n" podUID="6ecc2615-b458-41f8-839f-e22c53e7ad0e" Nov 03 01:17:26.359096 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.359077 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dfr9t" Nov 03 01:17:26.359201 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:17:26.359145 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dfr9t" podUID="6179266f-07ff-48cd-bf03-3f1e961529d7" Nov 03 01:17:26.360614 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.360595 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2jrfl" Nov 03 01:17:26.361978 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.361955 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-jpcqz" Nov 03 01:17:26.362683 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.362654 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/acada406-fef9-4a03-ae21-c2351ef0a4cf-tmp-dir\") pod \"node-resolver-5qr9g\" (UID: \"acada406-fef9-4a03-ae21-c2351ef0a4cf\") " pod="openshift-dns/node-resolver-5qr9g" Nov 03 01:17:26.362781 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.362706 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rwtq\" (UniqueName: \"kubernetes.io/projected/acada406-fef9-4a03-ae21-c2351ef0a4cf-kube-api-access-9rwtq\") pod \"node-resolver-5qr9g\" (UID: \"acada406-fef9-4a03-ae21-c2351ef0a4cf\") " pod="openshift-dns/node-resolver-5qr9g" Nov 03 01:17:26.362781 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.362742 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/abdd79d9-8d87-4133-85a2-643d26fc9faa-systemd-units\") pod \"ovnkube-node-n2cjg\" (UID: \"abdd79d9-8d87-4133-85a2-643d26fc9faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cjg" Nov 03 01:17:26.362781 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.362769 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/abdd79d9-8d87-4133-85a2-643d26fc9faa-host-slash\") pod \"ovnkube-node-n2cjg\" (UID: \"abdd79d9-8d87-4133-85a2-643d26fc9faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cjg" Nov 03 01:17:26.362895 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.362799 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/abdd79d9-8d87-4133-85a2-643d26fc9faa-run-systemd\") pod \"ovnkube-node-n2cjg\" (UID: \"abdd79d9-8d87-4133-85a2-643d26fc9faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cjg" Nov 03 01:17:26.362895 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.362829 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/abdd79d9-8d87-4133-85a2-643d26fc9faa-var-lib-openvswitch\") pod \"ovnkube-node-n2cjg\" (UID: \"abdd79d9-8d87-4133-85a2-643d26fc9faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cjg" Nov 03 01:17:26.362895 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.362860 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/abdd79d9-8d87-4133-85a2-643d26fc9faa-node-log\") pod \"ovnkube-node-n2cjg\" (UID: \"abdd79d9-8d87-4133-85a2-643d26fc9faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cjg" Nov 03 01:17:26.363034 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.363018 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/abdd79d9-8d87-4133-85a2-643d26fc9faa-host-cni-netd\") pod \"ovnkube-node-n2cjg\" (UID: \"abdd79d9-8d87-4133-85a2-643d26fc9faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cjg" Nov 03 01:17:26.363076 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.363061 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/acada406-fef9-4a03-ae21-c2351ef0a4cf-hosts-file\") pod \"node-resolver-5qr9g\" (UID: \"acada406-fef9-4a03-ae21-c2351ef0a4cf\") " pod="openshift-dns/node-resolver-5qr9g" Nov 03 01:17:26.363252 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.363131 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/abdd79d9-8d87-4133-85a2-643d26fc9faa-host-kubelet\") pod \"ovnkube-node-n2cjg\" (UID: \"abdd79d9-8d87-4133-85a2-643d26fc9faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cjg" Nov 03 01:17:26.363419 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.363400 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/abdd79d9-8d87-4133-85a2-643d26fc9faa-host-run-netns\") pod \"ovnkube-node-n2cjg\" (UID: \"abdd79d9-8d87-4133-85a2-643d26fc9faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cjg" Nov 03 01:17:26.363509 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.363485 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/abdd79d9-8d87-4133-85a2-643d26fc9faa-etc-openvswitch\") pod \"ovnkube-node-n2cjg\" (UID: \"abdd79d9-8d87-4133-85a2-643d26fc9faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cjg" Nov 03 01:17:26.363683 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.363660 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/abdd79d9-8d87-4133-85a2-643d26fc9faa-run-openvswitch\") pod \"ovnkube-node-n2cjg\" (UID: \"abdd79d9-8d87-4133-85a2-643d26fc9faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cjg" Nov 03 01:17:26.363741 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.363716 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/abdd79d9-8d87-4133-85a2-643d26fc9faa-log-socket\") pod \"ovnkube-node-n2cjg\" (UID: \"abdd79d9-8d87-4133-85a2-643d26fc9faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cjg" Nov 03 01:17:26.363803 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.363770 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/abdd79d9-8d87-4133-85a2-643d26fc9faa-host-run-ovn-kubernetes\") pod \"ovnkube-node-n2cjg\" (UID: \"abdd79d9-8d87-4133-85a2-643d26fc9faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cjg" Nov 03 01:17:26.363858 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.363828 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/abdd79d9-8d87-4133-85a2-643d26fc9faa-host-cni-bin\") pod \"ovnkube-node-n2cjg\" (UID: \"abdd79d9-8d87-4133-85a2-643d26fc9faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cjg" Nov 03 01:17:26.363909 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.363874 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/abdd79d9-8d87-4133-85a2-643d26fc9faa-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-n2cjg\" (UID: \"abdd79d9-8d87-4133-85a2-643d26fc9faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cjg" Nov 03 01:17:26.363961 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.363923 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/abdd79d9-8d87-4133-85a2-643d26fc9faa-ovnkube-config\") pod \"ovnkube-node-n2cjg\" (UID: \"abdd79d9-8d87-4133-85a2-643d26fc9faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cjg" Nov 03 01:17:26.364026 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.363956 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/abdd79d9-8d87-4133-85a2-643d26fc9faa-env-overrides\") pod \"ovnkube-node-n2cjg\" (UID: \"abdd79d9-8d87-4133-85a2-643d26fc9faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cjg" Nov 03 01:17:26.364087 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.364030 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-44p7d" Nov 03 01:17:26.364141 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.364029 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/abdd79d9-8d87-4133-85a2-643d26fc9faa-run-ovn\") pod \"ovnkube-node-n2cjg\" (UID: \"abdd79d9-8d87-4133-85a2-643d26fc9faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cjg" Nov 03 01:17:26.364193 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.364147 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/abdd79d9-8d87-4133-85a2-643d26fc9faa-ovn-node-metrics-cert\") pod \"ovnkube-node-n2cjg\" (UID: \"abdd79d9-8d87-4133-85a2-643d26fc9faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cjg" Nov 03 01:17:26.364193 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.364179 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/abdd79d9-8d87-4133-85a2-643d26fc9faa-ovnkube-script-lib\") pod \"ovnkube-node-n2cjg\" (UID: \"abdd79d9-8d87-4133-85a2-643d26fc9faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cjg" Nov 03 01:17:26.364299 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.364230 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9j8g\" (UniqueName: \"kubernetes.io/projected/abdd79d9-8d87-4133-85a2-643d26fc9faa-kube-api-access-h9j8g\") pod \"ovnkube-node-n2cjg\" (UID: \"abdd79d9-8d87-4133-85a2-643d26fc9faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cjg" Nov 03 01:17:26.366035 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.366011 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-x4tvh" Nov 03 01:17:26.407755 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.407727 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-r4kvh\"" Nov 03 01:17:26.407755 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.407727 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 03 01:17:26.407920 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.407772 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 03 01:17:26.408455 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.408443 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 03 01:17:26.408538 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.408518 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 03 01:17:26.408538 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.408528 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-c92gz\"" Nov 03 01:17:26.408658 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.408606 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 03 01:17:26.408822 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.408805 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 03 01:17:26.408881 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.408869 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 03 01:17:26.409250 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.409232 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 03 01:17:26.409319 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.409232 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 03 01:17:26.409704 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.409689 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 03 01:17:26.409704 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.409695 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-bdhgr\"" Nov 03 01:17:26.433452 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.433423 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-tcxvw\"" Nov 03 01:17:26.434031 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.434014 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 03 01:17:26.434412 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.434401 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 03 01:17:26.435638 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.435620 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 03 01:17:26.435902 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.435886 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 03 01:17:26.435962 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.435941 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 03 01:17:26.435962 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.435940 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 03 01:17:26.436059 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.435961 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 03 01:17:26.436059 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.435984 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-nkj8v\"" Nov 03 01:17:26.436508 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.436496 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 03 01:17:26.436836 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.436823 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 03 01:17:26.436960 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.436949 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 03 01:17:26.451620 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.451595 2578 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 03 01:17:26.464024 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.464003 2578 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 03 01:17:26.464125 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.464109 2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-n6dt8" Nov 03 01:17:26.464349 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.464330 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/abdd79d9-8d87-4133-85a2-643d26fc9faa-run-ovn\") pod \"ovnkube-node-n2cjg\" (UID: \"abdd79d9-8d87-4133-85a2-643d26fc9faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cjg" Nov 03 01:17:26.464469 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.464359 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e17d6c62-2952-420c-beda-49db831c48ac-host-var-lib-cni-multus\") pod \"multus-44p7d\" (UID: \"e17d6c62-2952-420c-beda-49db831c48ac\") " pod="openshift-multus/multus-44p7d" Nov 03 01:17:26.464469 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.464393 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e17d6c62-2952-420c-beda-49db831c48ac-host-var-lib-kubelet\") pod \"multus-44p7d\" (UID: \"e17d6c62-2952-420c-beda-49db831c48ac\") " pod="openshift-multus/multus-44p7d" Nov 03 01:17:26.464469 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.464414 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e17d6c62-2952-420c-beda-49db831c48ac-multus-conf-dir\") pod \"multus-44p7d\" (UID: \"e17d6c62-2952-420c-beda-49db831c48ac\") " pod="openshift-multus/multus-44p7d" Nov 03 01:17:26.464469 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.464432 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/abdd79d9-8d87-4133-85a2-643d26fc9faa-run-ovn\") pod \"ovnkube-node-n2cjg\" (UID: \"abdd79d9-8d87-4133-85a2-643d26fc9faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cjg" Nov 03 01:17:26.464621 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.464520 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fe434c5e-450b-4512-93aa-78d3e56e5002-konnectivity-ca\") pod \"konnectivity-agent-dd9jt\" (UID: \"fe434c5e-450b-4512-93aa-78d3e56e5002\") " pod="kube-system/konnectivity-agent-dd9jt" Nov 03 01:17:26.464621 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.464546 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2r8s\" (UniqueName: \"kubernetes.io/projected/20152fc3-0b5f-4c58-94fb-e02094377c74-kube-api-access-m2r8s\") pod \"iptables-alerter-x4tvh\" (UID: \"20152fc3-0b5f-4c58-94fb-e02094377c74\") " pod="openshift-network-operator/iptables-alerter-x4tvh" Nov 03 01:17:26.464621 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.464563 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d157814d-82d6-4af5-a648-c03ad35f3a41-tmp\") pod \"tuned-c6r2x\" (UID: \"d157814d-82d6-4af5-a648-c03ad35f3a41\") " pod="openshift-cluster-node-tuning-operator/tuned-c6r2x" Nov 03 01:17:26.464621 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.464582 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/acada406-fef9-4a03-ae21-c2351ef0a4cf-tmp-dir\") pod \"node-resolver-5qr9g\" (UID: \"acada406-fef9-4a03-ae21-c2351ef0a4cf\") " pod="openshift-dns/node-resolver-5qr9g" Nov 03 01:17:26.464808 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.464621 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/abdd79d9-8d87-4133-85a2-643d26fc9faa-host-cni-netd\") pod \"ovnkube-node-n2cjg\" (UID: \"abdd79d9-8d87-4133-85a2-643d26fc9faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cjg" Nov 03 01:17:26.464808 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.464694 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/abdd79d9-8d87-4133-85a2-643d26fc9faa-host-cni-netd\") pod \"ovnkube-node-n2cjg\" (UID: \"abdd79d9-8d87-4133-85a2-643d26fc9faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cjg" Nov 03 01:17:26.464808 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.464694 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bff481c2-d05e-4b45-827d-f5c8a0f60d0b-host\") pod \"node-ca-jpcqz\" (UID: \"bff481c2-d05e-4b45-827d-f5c8a0f60d0b\") " pod="openshift-image-registry/node-ca-jpcqz" Nov 03 01:17:26.464808 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.464722 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d157814d-82d6-4af5-a648-c03ad35f3a41-etc-kubernetes\") pod \"tuned-c6r2x\" (UID: \"d157814d-82d6-4af5-a648-c03ad35f3a41\") " pod="openshift-cluster-node-tuning-operator/tuned-c6r2x" Nov 03 01:17:26.464808 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.464756 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgd8w\" (UniqueName: \"kubernetes.io/projected/3b2bf2f6-28aa-4480-9f4a-f4e7f659e966-kube-api-access-vgd8w\") pod \"network-check-target-kg99x\" (UID: \"3b2bf2f6-28aa-4480-9f4a-f4e7f659e966\") " pod="openshift-network-diagnostics/network-check-target-kg99x" Nov 03 01:17:26.465037 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.464796 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/acada406-fef9-4a03-ae21-c2351ef0a4cf-hosts-file\") pod \"node-resolver-5qr9g\" (UID: \"acada406-fef9-4a03-ae21-c2351ef0a4cf\") " pod="openshift-dns/node-resolver-5qr9g" Nov 03 01:17:26.465037 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.464838 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/abdd79d9-8d87-4133-85a2-643d26fc9faa-etc-openvswitch\") pod \"ovnkube-node-n2cjg\" (UID: \"abdd79d9-8d87-4133-85a2-643d26fc9faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cjg" Nov 03 01:17:26.465037 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.464860 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/abdd79d9-8d87-4133-85a2-643d26fc9faa-env-overrides\") pod \"ovnkube-node-n2cjg\" (UID: \"abdd79d9-8d87-4133-85a2-643d26fc9faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cjg" Nov 03 01:17:26.465037 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.464885 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e17d6c62-2952-420c-beda-49db831c48ac-system-cni-dir\") pod \"multus-44p7d\" (UID: \"e17d6c62-2952-420c-beda-49db831c48ac\") " pod="openshift-multus/multus-44p7d" Nov 03 01:17:26.465037 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.464897 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/acada406-fef9-4a03-ae21-c2351ef0a4cf-hosts-file\") pod \"node-resolver-5qr9g\" (UID: \"acada406-fef9-4a03-ae21-c2351ef0a4cf\") " pod="openshift-dns/node-resolver-5qr9g" Nov 03 01:17:26.465037 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.464952 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/acada406-fef9-4a03-ae21-c2351ef0a4cf-tmp-dir\") pod \"node-resolver-5qr9g\" (UID: \"acada406-fef9-4a03-ae21-c2351ef0a4cf\") " pod="openshift-dns/node-resolver-5qr9g" Nov 03 01:17:26.465037 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.464964 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/abdd79d9-8d87-4133-85a2-643d26fc9faa-etc-openvswitch\") pod \"ovnkube-node-n2cjg\" (UID: \"abdd79d9-8d87-4133-85a2-643d26fc9faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cjg" Nov 03 01:17:26.465037 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.464974 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/20152fc3-0b5f-4c58-94fb-e02094377c74-host-slash\") pod \"iptables-alerter-x4tvh\" (UID: \"20152fc3-0b5f-4c58-94fb-e02094377c74\") " pod="openshift-network-operator/iptables-alerter-x4tvh" Nov 03 01:17:26.465037 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.465034 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cct5g\" (UniqueName: \"kubernetes.io/projected/d157814d-82d6-4af5-a648-c03ad35f3a41-kube-api-access-cct5g\") pod \"tuned-c6r2x\" (UID: \"d157814d-82d6-4af5-a648-c03ad35f3a41\") " pod="openshift-cluster-node-tuning-operator/tuned-c6r2x" Nov 03 01:17:26.465412 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.465061 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e17d6c62-2952-420c-beda-49db831c48ac-host-run-multus-certs\") pod \"multus-44p7d\" (UID: \"e17d6c62-2952-420c-beda-49db831c48ac\") " pod="openshift-multus/multus-44p7d" Nov 03 01:17:26.465412 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.465114 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ecc2615-b458-41f8-839f-e22c53e7ad0e-metrics-certs\") pod \"network-metrics-daemon-8g56n\" (UID: \"6ecc2615-b458-41f8-839f-e22c53e7ad0e\") " pod="openshift-multus/network-metrics-daemon-8g56n" Nov 03 01:17:26.465412 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.465164 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7skt\" (UniqueName: \"kubernetes.io/projected/e17d6c62-2952-420c-beda-49db831c48ac-kube-api-access-g7skt\") pod \"multus-44p7d\" (UID: \"e17d6c62-2952-420c-beda-49db831c48ac\") " pod="openshift-multus/multus-44p7d" Nov 03 01:17:26.465412 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.465196 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1febd8fa-5641-449b-b1fd-c7d02603fd1c-registration-dir\") pod \"aws-ebs-csi-driver-node-2jrfl\" (UID: \"1febd8fa-5641-449b-b1fd-c7d02603fd1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2jrfl" Nov 03 01:17:26.465412 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.465228 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d157814d-82d6-4af5-a648-c03ad35f3a41-etc-sysctl-conf\") pod \"tuned-c6r2x\" (UID: \"d157814d-82d6-4af5-a648-c03ad35f3a41\") " pod="openshift-cluster-node-tuning-operator/tuned-c6r2x" Nov 03 01:17:26.465412 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.465251 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d157814d-82d6-4af5-a648-c03ad35f3a41-lib-modules\") pod \"tuned-c6r2x\" (UID: \"d157814d-82d6-4af5-a648-c03ad35f3a41\") " pod="openshift-cluster-node-tuning-operator/tuned-c6r2x" Nov 03 01:17:26.465412 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.465270 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4c25011b-c85c-4736-9f72-842a96949072-cni-binary-copy\") pod \"multus-additional-cni-plugins-jnvg6\" (UID: \"4c25011b-c85c-4736-9f72-842a96949072\") " pod="openshift-multus/multus-additional-cni-plugins-jnvg6" Nov 03 01:17:26.465412 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.465254 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/abdd79d9-8d87-4133-85a2-643d26fc9faa-env-overrides\") pod \"ovnkube-node-n2cjg\" (UID: \"abdd79d9-8d87-4133-85a2-643d26fc9faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cjg" Nov 03 01:17:26.465412 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.465287 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e17d6c62-2952-420c-beda-49db831c48ac-cni-binary-copy\") pod \"multus-44p7d\" (UID: \"e17d6c62-2952-420c-beda-49db831c48ac\") " pod="openshift-multus/multus-44p7d" Nov 03 01:17:26.465412 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.465306 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e17d6c62-2952-420c-beda-49db831c48ac-multus-socket-dir-parent\") pod \"multus-44p7d\" (UID: \"e17d6c62-2952-420c-beda-49db831c48ac\") " pod="openshift-multus/multus-44p7d" Nov 03 01:17:26.465412 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.465321 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d157814d-82d6-4af5-a648-c03ad35f3a41-etc-systemd\") pod \"tuned-c6r2x\" (UID: \"d157814d-82d6-4af5-a648-c03ad35f3a41\") " pod="openshift-cluster-node-tuning-operator/tuned-c6r2x" Nov 03 01:17:26.465412 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.465377 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzr84\" (UniqueName: \"kubernetes.io/projected/4c25011b-c85c-4736-9f72-842a96949072-kube-api-access-vzr84\") pod \"multus-additional-cni-plugins-jnvg6\" (UID: \"4c25011b-c85c-4736-9f72-842a96949072\") " pod="openshift-multus/multus-additional-cni-plugins-jnvg6" Nov 03 01:17:26.465412 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.465416 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/abdd79d9-8d87-4133-85a2-643d26fc9faa-run-openvswitch\") pod \"ovnkube-node-n2cjg\" (UID: \"abdd79d9-8d87-4133-85a2-643d26fc9faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cjg" Nov 03 01:17:26.466068 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.465455 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e17d6c62-2952-420c-beda-49db831c48ac-multus-cni-dir\") pod \"multus-44p7d\" (UID: \"e17d6c62-2952-420c-beda-49db831c48ac\") " pod="openshift-multus/multus-44p7d" Nov 03 01:17:26.466068 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.465501 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/abdd79d9-8d87-4133-85a2-643d26fc9faa-run-openvswitch\") pod \"ovnkube-node-n2cjg\" (UID: \"abdd79d9-8d87-4133-85a2-643d26fc9faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cjg" Nov 03 01:17:26.466068 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.465523 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88c5l\" (UniqueName: \"kubernetes.io/projected/bff481c2-d05e-4b45-827d-f5c8a0f60d0b-kube-api-access-88c5l\") pod \"node-ca-jpcqz\" (UID: \"bff481c2-d05e-4b45-827d-f5c8a0f60d0b\") " pod="openshift-image-registry/node-ca-jpcqz" Nov 03 01:17:26.466068 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.465549 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/abdd79d9-8d87-4133-85a2-643d26fc9faa-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-n2cjg\" (UID: \"abdd79d9-8d87-4133-85a2-643d26fc9faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cjg" Nov 03 01:17:26.466068 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.465574 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/abdd79d9-8d87-4133-85a2-643d26fc9faa-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-n2cjg\" (UID: \"abdd79d9-8d87-4133-85a2-643d26fc9faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cjg" Nov 03 01:17:26.466068 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.465606 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b56kh\" (UniqueName: \"kubernetes.io/projected/6ecc2615-b458-41f8-839f-e22c53e7ad0e-kube-api-access-b56kh\") pod \"network-metrics-daemon-8g56n\" (UID: \"6ecc2615-b458-41f8-839f-e22c53e7ad0e\") " pod="openshift-multus/network-metrics-daemon-8g56n" Nov 03 01:17:26.466068 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.465631 2578 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 01:17:26.466068 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.465640 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d157814d-82d6-4af5-a648-c03ad35f3a41-sys\") pod \"tuned-c6r2x\" (UID: \"d157814d-82d6-4af5-a648-c03ad35f3a41\") " pod="openshift-cluster-node-tuning-operator/tuned-c6r2x" Nov 03 01:17:26.466068 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.465668 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d157814d-82d6-4af5-a648-c03ad35f3a41-host\") pod \"tuned-c6r2x\" (UID: \"d157814d-82d6-4af5-a648-c03ad35f3a41\") " pod="openshift-cluster-node-tuning-operator/tuned-c6r2x" Nov 03 01:17:26.466068 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.465694 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4c25011b-c85c-4736-9f72-842a96949072-system-cni-dir\") pod \"multus-additional-cni-plugins-jnvg6\" (UID: \"4c25011b-c85c-4736-9f72-842a96949072\") " pod="openshift-multus/multus-additional-cni-plugins-jnvg6" Nov 03 01:17:26.466068 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.465717 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/abdd79d9-8d87-4133-85a2-643d26fc9faa-ovn-node-metrics-cert\") pod \"ovnkube-node-n2cjg\" (UID: \"abdd79d9-8d87-4133-85a2-643d26fc9faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cjg" Nov 03 01:17:26.466068 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.465740 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e17d6c62-2952-420c-beda-49db831c48ac-etc-kubernetes\") pod \"multus-44p7d\" (UID: \"e17d6c62-2952-420c-beda-49db831c48ac\") " pod="openshift-multus/multus-44p7d" Nov 03 01:17:26.466068 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.465758 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9j8g\" (UniqueName: \"kubernetes.io/projected/abdd79d9-8d87-4133-85a2-643d26fc9faa-kube-api-access-h9j8g\") pod \"ovnkube-node-n2cjg\" (UID: \"abdd79d9-8d87-4133-85a2-643d26fc9faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cjg" Nov 03 01:17:26.466068 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.465775 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rwtq\" (UniqueName: \"kubernetes.io/projected/acada406-fef9-4a03-ae21-c2351ef0a4cf-kube-api-access-9rwtq\") pod \"node-resolver-5qr9g\" (UID: \"acada406-fef9-4a03-ae21-c2351ef0a4cf\") " pod="openshift-dns/node-resolver-5qr9g" Nov 03 01:17:26.466068 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.465792 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/abdd79d9-8d87-4133-85a2-643d26fc9faa-host-slash\") pod \"ovnkube-node-n2cjg\" (UID: \"abdd79d9-8d87-4133-85a2-643d26fc9faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cjg" Nov 03 01:17:26.466068 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.465809 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e17d6c62-2952-420c-beda-49db831c48ac-host-run-k8s-cni-cncf-io\") pod \"multus-44p7d\" (UID: \"e17d6c62-2952-420c-beda-49db831c48ac\") " pod="openshift-multus/multus-44p7d" Nov 03 01:17:26.466068 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.465834 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e17d6c62-2952-420c-beda-49db831c48ac-multus-daemon-config\") pod \"multus-44p7d\" (UID: \"e17d6c62-2952-420c-beda-49db831c48ac\") " pod="openshift-multus/multus-44p7d" Nov 03 01:17:26.466793 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.465899 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/abdd79d9-8d87-4133-85a2-643d26fc9faa-host-slash\") pod \"ovnkube-node-n2cjg\" (UID: \"abdd79d9-8d87-4133-85a2-643d26fc9faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cjg" Nov 03 01:17:26.466793 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.465902 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdm2b\" (UniqueName: \"kubernetes.io/projected/1febd8fa-5641-449b-b1fd-c7d02603fd1c-kube-api-access-bdm2b\") pod \"aws-ebs-csi-driver-node-2jrfl\" (UID: \"1febd8fa-5641-449b-b1fd-c7d02603fd1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2jrfl" Nov 03 01:17:26.466793 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.465938 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d157814d-82d6-4af5-a648-c03ad35f3a41-etc-tuned\") pod \"tuned-c6r2x\" (UID: \"d157814d-82d6-4af5-a648-c03ad35f3a41\") " pod="openshift-cluster-node-tuning-operator/tuned-c6r2x" Nov 03 01:17:26.466793 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.466008 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4c25011b-c85c-4736-9f72-842a96949072-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jnvg6\" (UID: \"4c25011b-c85c-4736-9f72-842a96949072\") " pod="openshift-multus/multus-additional-cni-plugins-jnvg6" Nov 03 01:17:26.466793 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.466047 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/abdd79d9-8d87-4133-85a2-643d26fc9faa-host-kubelet\") pod \"ovnkube-node-n2cjg\" (UID: \"abdd79d9-8d87-4133-85a2-643d26fc9faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cjg" Nov 03 01:17:26.466793 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.466055 2578 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 03 01:17:26.466793 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.466115 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/abdd79d9-8d87-4133-85a2-643d26fc9faa-host-run-ovn-kubernetes\") pod \"ovnkube-node-n2cjg\" (UID: \"abdd79d9-8d87-4133-85a2-643d26fc9faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cjg" Nov 03 01:17:26.466793 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.466076 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/abdd79d9-8d87-4133-85a2-643d26fc9faa-host-run-ovn-kubernetes\") pod \"ovnkube-node-n2cjg\" (UID: \"abdd79d9-8d87-4133-85a2-643d26fc9faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cjg" Nov 03 01:17:26.466793 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.466141 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/abdd79d9-8d87-4133-85a2-643d26fc9faa-host-kubelet\") pod \"ovnkube-node-n2cjg\" (UID: \"abdd79d9-8d87-4133-85a2-643d26fc9faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cjg" Nov 03 01:17:26.466793 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.466159 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1febd8fa-5641-449b-b1fd-c7d02603fd1c-device-dir\") pod \"aws-ebs-csi-driver-node-2jrfl\" (UID: \"1febd8fa-5641-449b-b1fd-c7d02603fd1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2jrfl" Nov 03 01:17:26.466793 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.466177 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1febd8fa-5641-449b-b1fd-c7d02603fd1c-sys-fs\") pod \"aws-ebs-csi-driver-node-2jrfl\" (UID: \"1febd8fa-5641-449b-b1fd-c7d02603fd1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2jrfl" Nov 03 01:17:26.466793 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.466213 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/abdd79d9-8d87-4133-85a2-643d26fc9faa-host-cni-bin\") pod \"ovnkube-node-n2cjg\" (UID: \"abdd79d9-8d87-4133-85a2-643d26fc9faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cjg" Nov 03 01:17:26.466793 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.466229 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e17d6c62-2952-420c-beda-49db831c48ac-cnibin\") pod \"multus-44p7d\" (UID: \"e17d6c62-2952-420c-beda-49db831c48ac\") " pod="openshift-multus/multus-44p7d" Nov 03 01:17:26.466793 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.466263 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/abdd79d9-8d87-4133-85a2-643d26fc9faa-host-cni-bin\") pod \"ovnkube-node-n2cjg\" (UID: \"abdd79d9-8d87-4133-85a2-643d26fc9faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cjg" Nov 03 01:17:26.466793 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.466287 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/bff481c2-d05e-4b45-827d-f5c8a0f60d0b-serviceca\") pod \"node-ca-jpcqz\" (UID: \"bff481c2-d05e-4b45-827d-f5c8a0f60d0b\") " pod="openshift-image-registry/node-ca-jpcqz" Nov 03 01:17:26.466793 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.466308 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4c25011b-c85c-4736-9f72-842a96949072-cnibin\") pod \"multus-additional-cni-plugins-jnvg6\" (UID: \"4c25011b-c85c-4736-9f72-842a96949072\") " pod="openshift-multus/multus-additional-cni-plugins-jnvg6" Nov 03 01:17:26.466793 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.466347 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/abdd79d9-8d87-4133-85a2-643d26fc9faa-ovnkube-script-lib\") pod \"ovnkube-node-n2cjg\" (UID: \"abdd79d9-8d87-4133-85a2-643d26fc9faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cjg" Nov 03 01:17:26.467474 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.466364 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e17d6c62-2952-420c-beda-49db831c48ac-host-run-netns\") pod \"multus-44p7d\" (UID: \"e17d6c62-2952-420c-beda-49db831c48ac\") " pod="openshift-multus/multus-44p7d" Nov 03 01:17:26.467474 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.466428 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e17d6c62-2952-420c-beda-49db831c48ac-hostroot\") pod \"multus-44p7d\" (UID: \"e17d6c62-2952-420c-beda-49db831c48ac\") " pod="openshift-multus/multus-44p7d" Nov 03 01:17:26.467474 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.466454 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1febd8fa-5641-449b-b1fd-c7d02603fd1c-socket-dir\") pod \"aws-ebs-csi-driver-node-2jrfl\" (UID: \"1febd8fa-5641-449b-b1fd-c7d02603fd1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2jrfl" Nov 03 01:17:26.467474 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.466476 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/20152fc3-0b5f-4c58-94fb-e02094377c74-iptables-alerter-script\") pod \"iptables-alerter-x4tvh\" (UID: \"20152fc3-0b5f-4c58-94fb-e02094377c74\") " pod="openshift-network-operator/iptables-alerter-x4tvh" Nov 03 01:17:26.467474 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.466536 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d157814d-82d6-4af5-a648-c03ad35f3a41-etc-sysctl-d\") pod \"tuned-c6r2x\" (UID: \"d157814d-82d6-4af5-a648-c03ad35f3a41\") " pod="openshift-cluster-node-tuning-operator/tuned-c6r2x" Nov 03 01:17:26.467474 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.466587 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d157814d-82d6-4af5-a648-c03ad35f3a41-var-lib-kubelet\") pod \"tuned-c6r2x\" (UID: \"d157814d-82d6-4af5-a648-c03ad35f3a41\") " pod="openshift-cluster-node-tuning-operator/tuned-c6r2x" Nov 03 01:17:26.467474 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.466624 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4c25011b-c85c-4736-9f72-842a96949072-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jnvg6\" (UID: \"4c25011b-c85c-4736-9f72-842a96949072\") " pod="openshift-multus/multus-additional-cni-plugins-jnvg6" Nov 03 01:17:26.467474 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.466649 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6179266f-07ff-48cd-bf03-3f1e961529d7-dbus\") pod \"global-pull-secret-syncer-dfr9t\" (UID: \"6179266f-07ff-48cd-bf03-3f1e961529d7\") " pod="kube-system/global-pull-secret-syncer-dfr9t" Nov 03 01:17:26.467474 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.466700 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/abdd79d9-8d87-4133-85a2-643d26fc9faa-systemd-units\") pod \"ovnkube-node-n2cjg\" (UID: \"abdd79d9-8d87-4133-85a2-643d26fc9faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cjg" Nov 03 01:17:26.467474 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.466736 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/abdd79d9-8d87-4133-85a2-643d26fc9faa-run-systemd\") pod \"ovnkube-node-n2cjg\" (UID: \"abdd79d9-8d87-4133-85a2-643d26fc9faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cjg" Nov 03 01:17:26.467474 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.466777 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/abdd79d9-8d87-4133-85a2-643d26fc9faa-systemd-units\") pod \"ovnkube-node-n2cjg\" (UID: \"abdd79d9-8d87-4133-85a2-643d26fc9faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cjg" Nov 03 01:17:26.467474 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.466788 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/abdd79d9-8d87-4133-85a2-643d26fc9faa-var-lib-openvswitch\") pod \"ovnkube-node-n2cjg\" (UID: \"abdd79d9-8d87-4133-85a2-643d26fc9faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cjg" Nov 03 01:17:26.467474 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.466812 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/abdd79d9-8d87-4133-85a2-643d26fc9faa-run-systemd\") pod \"ovnkube-node-n2cjg\" (UID: \"abdd79d9-8d87-4133-85a2-643d26fc9faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cjg" Nov 03 01:17:26.467474 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.466853 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/abdd79d9-8d87-4133-85a2-643d26fc9faa-node-log\") pod \"ovnkube-node-n2cjg\" (UID: \"abdd79d9-8d87-4133-85a2-643d26fc9faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cjg" Nov 03 01:17:26.467474 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.466877 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/abdd79d9-8d87-4133-85a2-643d26fc9faa-ovnkube-script-lib\") pod \"ovnkube-node-n2cjg\" (UID: \"abdd79d9-8d87-4133-85a2-643d26fc9faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cjg" Nov 03 01:17:26.467474 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.466888 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/abdd79d9-8d87-4133-85a2-643d26fc9faa-node-log\") pod \"ovnkube-node-n2cjg\" (UID: \"abdd79d9-8d87-4133-85a2-643d26fc9faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cjg" Nov 03 01:17:26.467474 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.466881 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1febd8fa-5641-449b-b1fd-c7d02603fd1c-etc-selinux\") pod \"aws-ebs-csi-driver-node-2jrfl\" (UID: \"1febd8fa-5641-449b-b1fd-c7d02603fd1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2jrfl" Nov 03 01:17:26.468011 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.466890 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/abdd79d9-8d87-4133-85a2-643d26fc9faa-var-lib-openvswitch\") pod \"ovnkube-node-n2cjg\" (UID: \"abdd79d9-8d87-4133-85a2-643d26fc9faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cjg" Nov 03 01:17:26.468011 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.466938 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d157814d-82d6-4af5-a648-c03ad35f3a41-etc-modprobe-d\") pod \"tuned-c6r2x\" (UID: \"d157814d-82d6-4af5-a648-c03ad35f3a41\") " pod="openshift-cluster-node-tuning-operator/tuned-c6r2x" Nov 03 01:17:26.468011 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.466966 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d157814d-82d6-4af5-a648-c03ad35f3a41-etc-sysconfig\") pod \"tuned-c6r2x\" (UID: \"d157814d-82d6-4af5-a648-c03ad35f3a41\") " pod="openshift-cluster-node-tuning-operator/tuned-c6r2x" Nov 03 01:17:26.468011 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.466996 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/abdd79d9-8d87-4133-85a2-643d26fc9faa-host-run-netns\") pod \"ovnkube-node-n2cjg\" (UID: \"abdd79d9-8d87-4133-85a2-643d26fc9faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cjg" Nov 03 01:17:26.468011 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.467021 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/abdd79d9-8d87-4133-85a2-643d26fc9faa-log-socket\") pod \"ovnkube-node-n2cjg\" (UID: \"abdd79d9-8d87-4133-85a2-643d26fc9faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cjg" Nov 03 01:17:26.468011 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.467031 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/abdd79d9-8d87-4133-85a2-643d26fc9faa-host-run-netns\") pod \"ovnkube-node-n2cjg\" (UID: \"abdd79d9-8d87-4133-85a2-643d26fc9faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cjg" Nov 03 01:17:26.468011 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.467046 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/abdd79d9-8d87-4133-85a2-643d26fc9faa-ovnkube-config\") pod \"ovnkube-node-n2cjg\" (UID: \"abdd79d9-8d87-4133-85a2-643d26fc9faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cjg" Nov 03 01:17:26.468011 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.467054 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/abdd79d9-8d87-4133-85a2-643d26fc9faa-log-socket\") pod \"ovnkube-node-n2cjg\" (UID: \"abdd79d9-8d87-4133-85a2-643d26fc9faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cjg" Nov 03 01:17:26.468011 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.467074 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6179266f-07ff-48cd-bf03-3f1e961529d7-original-pull-secret\") pod \"global-pull-secret-syncer-dfr9t\" (UID: \"6179266f-07ff-48cd-bf03-3f1e961529d7\") " pod="kube-system/global-pull-secret-syncer-dfr9t" Nov 03 01:17:26.468011 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.467100 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fe434c5e-450b-4512-93aa-78d3e56e5002-agent-certs\") pod \"konnectivity-agent-dd9jt\" (UID: \"fe434c5e-450b-4512-93aa-78d3e56e5002\") " pod="kube-system/konnectivity-agent-dd9jt" Nov 03 01:17:26.468011 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.467123 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d157814d-82d6-4af5-a648-c03ad35f3a41-run\") pod \"tuned-c6r2x\" (UID: \"d157814d-82d6-4af5-a648-c03ad35f3a41\") " pod="openshift-cluster-node-tuning-operator/tuned-c6r2x" Nov 03 01:17:26.468011 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.467159 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4c25011b-c85c-4736-9f72-842a96949072-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jnvg6\" (UID: \"4c25011b-c85c-4736-9f72-842a96949072\") " pod="openshift-multus/multus-additional-cni-plugins-jnvg6" Nov 03 01:17:26.468011 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.467191 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6179266f-07ff-48cd-bf03-3f1e961529d7-kubelet-config\") pod \"global-pull-secret-syncer-dfr9t\" (UID: \"6179266f-07ff-48cd-bf03-3f1e961529d7\") " pod="kube-system/global-pull-secret-syncer-dfr9t" Nov 03 01:17:26.468011 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.467213 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e17d6c62-2952-420c-beda-49db831c48ac-host-var-lib-cni-bin\") pod \"multus-44p7d\" (UID: \"e17d6c62-2952-420c-beda-49db831c48ac\") " pod="openshift-multus/multus-44p7d" Nov 03 01:17:26.468011 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.467233 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1febd8fa-5641-449b-b1fd-c7d02603fd1c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2jrfl\" (UID: \"1febd8fa-5641-449b-b1fd-c7d02603fd1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2jrfl" Nov 03 01:17:26.468011 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.467485 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/abdd79d9-8d87-4133-85a2-643d26fc9faa-ovnkube-config\") pod \"ovnkube-node-n2cjg\" (UID: \"abdd79d9-8d87-4133-85a2-643d26fc9faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cjg" Nov 03 01:17:26.469718 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.469701 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/abdd79d9-8d87-4133-85a2-643d26fc9faa-ovn-node-metrics-cert\") pod \"ovnkube-node-n2cjg\" (UID: \"abdd79d9-8d87-4133-85a2-643d26fc9faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cjg" Nov 03 01:17:26.514875 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.514821 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-16.us-west-2.compute.internal" event={"ID":"f7e070a0c923c611d3cf96ab59dc8b57","Type":"ContainerStarted","Data":"a57cfab1072e0e4b06e4b4217747174d1f2639b037b9403811af934a57dc23e5"} Nov 03 01:17:26.515981 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.515947 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-16.us-west-2.compute.internal" event={"ID":"7f364a286e3c240d587086e0ec2180aa","Type":"ContainerStarted","Data":"640208824d01bacb81bae87037315951d0d6301e18043589626cbb13a7a78d2b"} Nov 03 01:17:26.516342 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.516323 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 03 01:17:26.516490 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.516403 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 03 01:17:26.516490 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.516407 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 03 01:17:26.516604 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.516585 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 03 01:17:26.516661 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.516609 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-th4bh\"" Nov 03 01:17:26.516661 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.516627 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-9tz7d\"" Nov 03 01:17:26.516661 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.516643 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-2fc25\"" Nov 03 01:17:26.516661 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.516613 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 03 01:17:26.516661 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.516654 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-t8fl2\"" Nov 03 01:17:26.516838 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.516667 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 03 01:17:26.516838 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.516697 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 03 01:17:26.533511 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.533491 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9j8g\" (UniqueName: \"kubernetes.io/projected/abdd79d9-8d87-4133-85a2-643d26fc9faa-kube-api-access-h9j8g\") pod \"ovnkube-node-n2cjg\" (UID: \"abdd79d9-8d87-4133-85a2-643d26fc9faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cjg" Nov 03 01:17:26.533600 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.533495 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rwtq\" (UniqueName: \"kubernetes.io/projected/acada406-fef9-4a03-ae21-c2351ef0a4cf-kube-api-access-9rwtq\") pod \"node-resolver-5qr9g\" (UID: \"acada406-fef9-4a03-ae21-c2351ef0a4cf\") " pod="openshift-dns/node-resolver-5qr9g" Nov 03 01:17:26.568422 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.568389 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d157814d-82d6-4af5-a648-c03ad35f3a41-host\") pod \"tuned-c6r2x\" (UID: \"d157814d-82d6-4af5-a648-c03ad35f3a41\") " pod="openshift-cluster-node-tuning-operator/tuned-c6r2x" Nov 03 01:17:26.568494 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.568442 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4c25011b-c85c-4736-9f72-842a96949072-system-cni-dir\") pod \"multus-additional-cni-plugins-jnvg6\" (UID: \"4c25011b-c85c-4736-9f72-842a96949072\") " pod="openshift-multus/multus-additional-cni-plugins-jnvg6" Nov 03 01:17:26.568494 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.568465 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e17d6c62-2952-420c-beda-49db831c48ac-etc-kubernetes\") pod \"multus-44p7d\" (UID: \"e17d6c62-2952-420c-beda-49db831c48ac\") " pod="openshift-multus/multus-44p7d" Nov 03 01:17:26.568494 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.568475 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d157814d-82d6-4af5-a648-c03ad35f3a41-host\") pod \"tuned-c6r2x\" (UID: \"d157814d-82d6-4af5-a648-c03ad35f3a41\") " pod="openshift-cluster-node-tuning-operator/tuned-c6r2x" Nov 03 01:17:26.568494 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.568483 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e17d6c62-2952-420c-beda-49db831c48ac-host-run-k8s-cni-cncf-io\") pod \"multus-44p7d\" (UID: \"e17d6c62-2952-420c-beda-49db831c48ac\") " pod="openshift-multus/multus-44p7d" Nov 03 01:17:26.568669 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.568499 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e17d6c62-2952-420c-beda-49db831c48ac-multus-daemon-config\") pod \"multus-44p7d\" (UID: \"e17d6c62-2952-420c-beda-49db831c48ac\") " pod="openshift-multus/multus-44p7d" Nov 03 01:17:26.568669 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.568529 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdm2b\" (UniqueName: \"kubernetes.io/projected/1febd8fa-5641-449b-b1fd-c7d02603fd1c-kube-api-access-bdm2b\") pod \"aws-ebs-csi-driver-node-2jrfl\" (UID: \"1febd8fa-5641-449b-b1fd-c7d02603fd1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2jrfl" Nov 03 01:17:26.568669 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.568535 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e17d6c62-2952-420c-beda-49db831c48ac-etc-kubernetes\") pod \"multus-44p7d\" (UID: \"e17d6c62-2952-420c-beda-49db831c48ac\") " pod="openshift-multus/multus-44p7d" Nov 03 01:17:26.568669 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.568547 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d157814d-82d6-4af5-a648-c03ad35f3a41-etc-tuned\") pod \"tuned-c6r2x\" (UID: \"d157814d-82d6-4af5-a648-c03ad35f3a41\") " pod="openshift-cluster-node-tuning-operator/tuned-c6r2x" Nov 03 01:17:26.568669 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.568559 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e17d6c62-2952-420c-beda-49db831c48ac-host-run-k8s-cni-cncf-io\") pod \"multus-44p7d\" (UID: \"e17d6c62-2952-420c-beda-49db831c48ac\") " pod="openshift-multus/multus-44p7d" Nov 03 01:17:26.568669 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.568567 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4c25011b-c85c-4736-9f72-842a96949072-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jnvg6\" (UID: \"4c25011b-c85c-4736-9f72-842a96949072\") " pod="openshift-multus/multus-additional-cni-plugins-jnvg6" Nov 03 01:17:26.568669 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.568602 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1febd8fa-5641-449b-b1fd-c7d02603fd1c-device-dir\") pod \"aws-ebs-csi-driver-node-2jrfl\" (UID: \"1febd8fa-5641-449b-b1fd-c7d02603fd1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2jrfl" Nov 03 01:17:26.568669 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.568627 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1febd8fa-5641-449b-b1fd-c7d02603fd1c-sys-fs\") pod \"aws-ebs-csi-driver-node-2jrfl\" (UID: \"1febd8fa-5641-449b-b1fd-c7d02603fd1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2jrfl" Nov 03 01:17:26.568669 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.568651 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e17d6c62-2952-420c-beda-49db831c48ac-cnibin\") pod \"multus-44p7d\" (UID: \"e17d6c62-2952-420c-beda-49db831c48ac\") " pod="openshift-multus/multus-44p7d" Nov 03 01:17:26.568669 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.568654 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4c25011b-c85c-4736-9f72-842a96949072-system-cni-dir\") pod \"multus-additional-cni-plugins-jnvg6\" (UID: \"4c25011b-c85c-4736-9f72-842a96949072\") " pod="openshift-multus/multus-additional-cni-plugins-jnvg6" Nov 03 01:17:26.569158 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.568677 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/bff481c2-d05e-4b45-827d-f5c8a0f60d0b-serviceca\") pod \"node-ca-jpcqz\" (UID: \"bff481c2-d05e-4b45-827d-f5c8a0f60d0b\") " pod="openshift-image-registry/node-ca-jpcqz" Nov 03 01:17:26.569158 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.568713 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4c25011b-c85c-4736-9f72-842a96949072-cnibin\") pod \"multus-additional-cni-plugins-jnvg6\" (UID: \"4c25011b-c85c-4736-9f72-842a96949072\") " pod="openshift-multus/multus-additional-cni-plugins-jnvg6" Nov 03 01:17:26.569158 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.568717 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e17d6c62-2952-420c-beda-49db831c48ac-cnibin\") pod \"multus-44p7d\" (UID: \"e17d6c62-2952-420c-beda-49db831c48ac\") " pod="openshift-multus/multus-44p7d" Nov 03 01:17:26.569158 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.568772 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4c25011b-c85c-4736-9f72-842a96949072-cnibin\") pod \"multus-additional-cni-plugins-jnvg6\" (UID: \"4c25011b-c85c-4736-9f72-842a96949072\") " pod="openshift-multus/multus-additional-cni-plugins-jnvg6" Nov 03 01:17:26.569158 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.568864 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e17d6c62-2952-420c-beda-49db831c48ac-host-run-netns\") pod \"multus-44p7d\" (UID: \"e17d6c62-2952-420c-beda-49db831c48ac\") " pod="openshift-multus/multus-44p7d" Nov 03 01:17:26.569158 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.569068 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1febd8fa-5641-449b-b1fd-c7d02603fd1c-sys-fs\") pod \"aws-ebs-csi-driver-node-2jrfl\" (UID: \"1febd8fa-5641-449b-b1fd-c7d02603fd1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2jrfl" Nov 03 01:17:26.569386 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.569228 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/bff481c2-d05e-4b45-827d-f5c8a0f60d0b-serviceca\") pod \"node-ca-jpcqz\" (UID: \"bff481c2-d05e-4b45-827d-f5c8a0f60d0b\") " pod="openshift-image-registry/node-ca-jpcqz" Nov 03 01:17:26.569386 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.569239 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e17d6c62-2952-420c-beda-49db831c48ac-multus-daemon-config\") pod \"multus-44p7d\" (UID: \"e17d6c62-2952-420c-beda-49db831c48ac\") " pod="openshift-multus/multus-44p7d" Nov 03 01:17:26.569386 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.568759 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e17d6c62-2952-420c-beda-49db831c48ac-host-run-netns\") pod \"multus-44p7d\" (UID: \"e17d6c62-2952-420c-beda-49db831c48ac\") " pod="openshift-multus/multus-44p7d" Nov 03 01:17:26.569386 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.569280 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1febd8fa-5641-449b-b1fd-c7d02603fd1c-device-dir\") pod \"aws-ebs-csi-driver-node-2jrfl\" (UID: \"1febd8fa-5641-449b-b1fd-c7d02603fd1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2jrfl" Nov 03 01:17:26.569386 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.569330 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e17d6c62-2952-420c-beda-49db831c48ac-hostroot\") pod \"multus-44p7d\" (UID: \"e17d6c62-2952-420c-beda-49db831c48ac\") " pod="openshift-multus/multus-44p7d" Nov 03 01:17:26.569386 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.569350 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1febd8fa-5641-449b-b1fd-c7d02603fd1c-socket-dir\") pod \"aws-ebs-csi-driver-node-2jrfl\" (UID: \"1febd8fa-5641-449b-b1fd-c7d02603fd1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2jrfl" Nov 03 01:17:26.569386 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.569372 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/20152fc3-0b5f-4c58-94fb-e02094377c74-iptables-alerter-script\") pod \"iptables-alerter-x4tvh\" (UID: \"20152fc3-0b5f-4c58-94fb-e02094377c74\") " pod="openshift-network-operator/iptables-alerter-x4tvh" Nov 03 01:17:26.569734 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.569400 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d157814d-82d6-4af5-a648-c03ad35f3a41-etc-sysctl-d\") pod \"tuned-c6r2x\" (UID: \"d157814d-82d6-4af5-a648-c03ad35f3a41\") " pod="openshift-cluster-node-tuning-operator/tuned-c6r2x" Nov 03 01:17:26.569734 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.569407 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e17d6c62-2952-420c-beda-49db831c48ac-hostroot\") pod \"multus-44p7d\" (UID: \"e17d6c62-2952-420c-beda-49db831c48ac\") " pod="openshift-multus/multus-44p7d" Nov 03 01:17:26.569734 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.569439 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d157814d-82d6-4af5-a648-c03ad35f3a41-var-lib-kubelet\") pod \"tuned-c6r2x\" (UID: \"d157814d-82d6-4af5-a648-c03ad35f3a41\") " pod="openshift-cluster-node-tuning-operator/tuned-c6r2x" Nov 03 01:17:26.569734 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.569474 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4c25011b-c85c-4736-9f72-842a96949072-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jnvg6\" (UID: \"4c25011b-c85c-4736-9f72-842a96949072\") " pod="openshift-multus/multus-additional-cni-plugins-jnvg6" Nov 03 01:17:26.569734 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.569489 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4c25011b-c85c-4736-9f72-842a96949072-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jnvg6\" (UID: \"4c25011b-c85c-4736-9f72-842a96949072\") " pod="openshift-multus/multus-additional-cni-plugins-jnvg6" Nov 03 01:17:26.569734 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.569501 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6179266f-07ff-48cd-bf03-3f1e961529d7-dbus\") pod \"global-pull-secret-syncer-dfr9t\" (UID: \"6179266f-07ff-48cd-bf03-3f1e961529d7\") " pod="kube-system/global-pull-secret-syncer-dfr9t" Nov 03 01:17:26.569734 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.569517 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1febd8fa-5641-449b-b1fd-c7d02603fd1c-socket-dir\") pod \"aws-ebs-csi-driver-node-2jrfl\" (UID: \"1febd8fa-5641-449b-b1fd-c7d02603fd1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2jrfl" Nov 03 01:17:26.569734 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.569561 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d157814d-82d6-4af5-a648-c03ad35f3a41-etc-sysctl-d\") pod \"tuned-c6r2x\" (UID: \"d157814d-82d6-4af5-a648-c03ad35f3a41\") " pod="openshift-cluster-node-tuning-operator/tuned-c6r2x" Nov 03 01:17:26.569734 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.569610 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1febd8fa-5641-449b-b1fd-c7d02603fd1c-etc-selinux\") pod \"aws-ebs-csi-driver-node-2jrfl\" (UID: \"1febd8fa-5641-449b-b1fd-c7d02603fd1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2jrfl" Nov 03 01:17:26.569734 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.569618 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d157814d-82d6-4af5-a648-c03ad35f3a41-var-lib-kubelet\") pod \"tuned-c6r2x\" (UID: \"d157814d-82d6-4af5-a648-c03ad35f3a41\") " pod="openshift-cluster-node-tuning-operator/tuned-c6r2x" Nov 03 01:17:26.569734 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.569632 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d157814d-82d6-4af5-a648-c03ad35f3a41-etc-modprobe-d\") pod \"tuned-c6r2x\" (UID: \"d157814d-82d6-4af5-a648-c03ad35f3a41\") " pod="openshift-cluster-node-tuning-operator/tuned-c6r2x" Nov 03 01:17:26.569734 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.569670 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d157814d-82d6-4af5-a648-c03ad35f3a41-etc-sysconfig\") pod \"tuned-c6r2x\" (UID: \"d157814d-82d6-4af5-a648-c03ad35f3a41\") " pod="openshift-cluster-node-tuning-operator/tuned-c6r2x" Nov 03 01:17:26.569734 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.569694 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6179266f-07ff-48cd-bf03-3f1e961529d7-original-pull-secret\") pod \"global-pull-secret-syncer-dfr9t\" (UID: \"6179266f-07ff-48cd-bf03-3f1e961529d7\") " pod="kube-system/global-pull-secret-syncer-dfr9t" Nov 03 01:17:26.569734 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.569710 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fe434c5e-450b-4512-93aa-78d3e56e5002-agent-certs\") pod \"konnectivity-agent-dd9jt\" (UID: \"fe434c5e-450b-4512-93aa-78d3e56e5002\") " pod="kube-system/konnectivity-agent-dd9jt" Nov 03 01:17:26.569734 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.569731 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d157814d-82d6-4af5-a648-c03ad35f3a41-run\") pod \"tuned-c6r2x\" (UID: \"d157814d-82d6-4af5-a648-c03ad35f3a41\") " pod="openshift-cluster-node-tuning-operator/tuned-c6r2x" Nov 03 01:17:26.569734 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.569737 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d157814d-82d6-4af5-a648-c03ad35f3a41-etc-modprobe-d\") pod \"tuned-c6r2x\" (UID: \"d157814d-82d6-4af5-a648-c03ad35f3a41\") " pod="openshift-cluster-node-tuning-operator/tuned-c6r2x" Nov 03 01:17:26.570553 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.569756 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4c25011b-c85c-4736-9f72-842a96949072-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jnvg6\" (UID: \"4c25011b-c85c-4736-9f72-842a96949072\") " pod="openshift-multus/multus-additional-cni-plugins-jnvg6" Nov 03 01:17:26.570553 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.569766 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d157814d-82d6-4af5-a648-c03ad35f3a41-etc-sysconfig\") pod \"tuned-c6r2x\" (UID: \"d157814d-82d6-4af5-a648-c03ad35f3a41\") " pod="openshift-cluster-node-tuning-operator/tuned-c6r2x" Nov 03 01:17:26.570553 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.569774 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6179266f-07ff-48cd-bf03-3f1e961529d7-dbus\") pod \"global-pull-secret-syncer-dfr9t\" (UID: \"6179266f-07ff-48cd-bf03-3f1e961529d7\") " pod="kube-system/global-pull-secret-syncer-dfr9t" Nov 03 01:17:26.570553 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.569670 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1febd8fa-5641-449b-b1fd-c7d02603fd1c-etc-selinux\") pod \"aws-ebs-csi-driver-node-2jrfl\" (UID: \"1febd8fa-5641-449b-b1fd-c7d02603fd1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2jrfl" Nov 03 01:17:26.570553 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.569784 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6179266f-07ff-48cd-bf03-3f1e961529d7-kubelet-config\") pod \"global-pull-secret-syncer-dfr9t\" (UID: \"6179266f-07ff-48cd-bf03-3f1e961529d7\") " pod="kube-system/global-pull-secret-syncer-dfr9t" Nov 03 01:17:26.570553 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.569830 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e17d6c62-2952-420c-beda-49db831c48ac-host-var-lib-cni-bin\") pod \"multus-44p7d\" (UID: \"e17d6c62-2952-420c-beda-49db831c48ac\") " pod="openshift-multus/multus-44p7d" Nov 03 01:17:26.570553 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.569835 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d157814d-82d6-4af5-a648-c03ad35f3a41-run\") pod \"tuned-c6r2x\" (UID: \"d157814d-82d6-4af5-a648-c03ad35f3a41\") " pod="openshift-cluster-node-tuning-operator/tuned-c6r2x" Nov 03 01:17:26.570553 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.569835 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6179266f-07ff-48cd-bf03-3f1e961529d7-kubelet-config\") pod \"global-pull-secret-syncer-dfr9t\" (UID: \"6179266f-07ff-48cd-bf03-3f1e961529d7\") " pod="kube-system/global-pull-secret-syncer-dfr9t" Nov 03 01:17:26.570553 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.569857 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1febd8fa-5641-449b-b1fd-c7d02603fd1c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2jrfl\" (UID: \"1febd8fa-5641-449b-b1fd-c7d02603fd1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2jrfl" Nov 03 01:17:26.570553 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.569880 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e17d6c62-2952-420c-beda-49db831c48ac-host-var-lib-cni-bin\") pod \"multus-44p7d\" (UID: \"e17d6c62-2952-420c-beda-49db831c48ac\") " pod="openshift-multus/multus-44p7d" Nov 03 01:17:26.570553 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.569889 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e17d6c62-2952-420c-beda-49db831c48ac-host-var-lib-cni-multus\") pod \"multus-44p7d\" (UID: \"e17d6c62-2952-420c-beda-49db831c48ac\") " pod="openshift-multus/multus-44p7d" Nov 03 01:17:26.570553 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.569914 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1febd8fa-5641-449b-b1fd-c7d02603fd1c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2jrfl\" (UID: \"1febd8fa-5641-449b-b1fd-c7d02603fd1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2jrfl" Nov 03 01:17:26.570553 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.569914 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e17d6c62-2952-420c-beda-49db831c48ac-host-var-lib-kubelet\") pod \"multus-44p7d\" (UID: \"e17d6c62-2952-420c-beda-49db831c48ac\") " pod="openshift-multus/multus-44p7d" Nov 03 01:17:26.570553 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.569937 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e17d6c62-2952-420c-beda-49db831c48ac-host-var-lib-kubelet\") pod \"multus-44p7d\" (UID: \"e17d6c62-2952-420c-beda-49db831c48ac\") " pod="openshift-multus/multus-44p7d" Nov 03 01:17:26.570553 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.569950 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e17d6c62-2952-420c-beda-49db831c48ac-multus-conf-dir\") pod \"multus-44p7d\" (UID: \"e17d6c62-2952-420c-beda-49db831c48ac\") " pod="openshift-multus/multus-44p7d" Nov 03 01:17:26.570553 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.569972 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e17d6c62-2952-420c-beda-49db831c48ac-host-var-lib-cni-multus\") pod \"multus-44p7d\" (UID: \"e17d6c62-2952-420c-beda-49db831c48ac\") " pod="openshift-multus/multus-44p7d" Nov 03 01:17:26.570553 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.569978 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fe434c5e-450b-4512-93aa-78d3e56e5002-konnectivity-ca\") pod \"konnectivity-agent-dd9jt\" (UID: \"fe434c5e-450b-4512-93aa-78d3e56e5002\") " pod="kube-system/konnectivity-agent-dd9jt" Nov 03 01:17:26.571413 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.570002 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4c25011b-c85c-4736-9f72-842a96949072-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jnvg6\" (UID: \"4c25011b-c85c-4736-9f72-842a96949072\") " pod="openshift-multus/multus-additional-cni-plugins-jnvg6" Nov 03 01:17:26.571413 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.570022 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/20152fc3-0b5f-4c58-94fb-e02094377c74-iptables-alerter-script\") pod \"iptables-alerter-x4tvh\" (UID: \"20152fc3-0b5f-4c58-94fb-e02094377c74\") " pod="openshift-network-operator/iptables-alerter-x4tvh" Nov 03 01:17:26.571413 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.570033 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e17d6c62-2952-420c-beda-49db831c48ac-multus-conf-dir\") pod \"multus-44p7d\" (UID: \"e17d6c62-2952-420c-beda-49db831c48ac\") " pod="openshift-multus/multus-44p7d" Nov 03 01:17:26.571413 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.570007 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2r8s\" (UniqueName: \"kubernetes.io/projected/20152fc3-0b5f-4c58-94fb-e02094377c74-kube-api-access-m2r8s\") pod \"iptables-alerter-x4tvh\" (UID: \"20152fc3-0b5f-4c58-94fb-e02094377c74\") " pod="openshift-network-operator/iptables-alerter-x4tvh" Nov 03 01:17:26.571413 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.570022 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4c25011b-c85c-4736-9f72-842a96949072-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jnvg6\" (UID: \"4c25011b-c85c-4736-9f72-842a96949072\") " pod="openshift-multus/multus-additional-cni-plugins-jnvg6" Nov 03 01:17:26.571413 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:17:26.570058 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 01:17:26.571413 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.570064 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d157814d-82d6-4af5-a648-c03ad35f3a41-tmp\") pod \"tuned-c6r2x\" (UID: \"d157814d-82d6-4af5-a648-c03ad35f3a41\") " pod="openshift-cluster-node-tuning-operator/tuned-c6r2x" Nov 03 01:17:26.571413 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.570088 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bff481c2-d05e-4b45-827d-f5c8a0f60d0b-host\") pod \"node-ca-jpcqz\" (UID: \"bff481c2-d05e-4b45-827d-f5c8a0f60d0b\") " pod="openshift-image-registry/node-ca-jpcqz" Nov 03 01:17:26.571413 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.570110 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d157814d-82d6-4af5-a648-c03ad35f3a41-etc-kubernetes\") pod \"tuned-c6r2x\" (UID: \"d157814d-82d6-4af5-a648-c03ad35f3a41\") " pod="openshift-cluster-node-tuning-operator/tuned-c6r2x" Nov 03 01:17:26.571413 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.570133 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bff481c2-d05e-4b45-827d-f5c8a0f60d0b-host\") pod \"node-ca-jpcqz\" (UID: \"bff481c2-d05e-4b45-827d-f5c8a0f60d0b\") " pod="openshift-image-registry/node-ca-jpcqz" Nov 03 01:17:26.571413 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.570135 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgd8w\" (UniqueName: \"kubernetes.io/projected/3b2bf2f6-28aa-4480-9f4a-f4e7f659e966-kube-api-access-vgd8w\") pod \"network-check-target-kg99x\" (UID: \"3b2bf2f6-28aa-4480-9f4a-f4e7f659e966\") " pod="openshift-network-diagnostics/network-check-target-kg99x" Nov 03 01:17:26.571413 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.570174 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e17d6c62-2952-420c-beda-49db831c48ac-system-cni-dir\") pod \"multus-44p7d\" (UID: \"e17d6c62-2952-420c-beda-49db831c48ac\") " pod="openshift-multus/multus-44p7d" Nov 03 01:17:26.571413 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.570201 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/20152fc3-0b5f-4c58-94fb-e02094377c74-host-slash\") pod \"iptables-alerter-x4tvh\" (UID: \"20152fc3-0b5f-4c58-94fb-e02094377c74\") " pod="openshift-network-operator/iptables-alerter-x4tvh" Nov 03 01:17:26.571413 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.570204 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d157814d-82d6-4af5-a648-c03ad35f3a41-etc-kubernetes\") pod \"tuned-c6r2x\" (UID: \"d157814d-82d6-4af5-a648-c03ad35f3a41\") " pod="openshift-cluster-node-tuning-operator/tuned-c6r2x" Nov 03 01:17:26.571413 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.570247 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/20152fc3-0b5f-4c58-94fb-e02094377c74-host-slash\") pod \"iptables-alerter-x4tvh\" (UID: \"20152fc3-0b5f-4c58-94fb-e02094377c74\") " pod="openshift-network-operator/iptables-alerter-x4tvh" Nov 03 01:17:26.571413 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:17:26.570261 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6179266f-07ff-48cd-bf03-3f1e961529d7-original-pull-secret podName:6179266f-07ff-48cd-bf03-3f1e961529d7 nodeName:}" failed. No retries permitted until 2026-11-03 01:17:27.070214255 +0000 UTC m=+3.157069821 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6179266f-07ff-48cd-bf03-3f1e961529d7-original-pull-secret") pod "global-pull-secret-syncer-dfr9t" (UID: "6179266f-07ff-48cd-bf03-3f1e961529d7") : object "kube-system"/"original-pull-secret" not registered Nov 03 01:17:26.571413 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.570284 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cct5g\" (UniqueName: \"kubernetes.io/projected/d157814d-82d6-4af5-a648-c03ad35f3a41-kube-api-access-cct5g\") pod \"tuned-c6r2x\" (UID: \"d157814d-82d6-4af5-a648-c03ad35f3a41\") " pod="openshift-cluster-node-tuning-operator/tuned-c6r2x" Nov 03 01:17:26.572286 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.570315 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e17d6c62-2952-420c-beda-49db831c48ac-host-run-multus-certs\") pod \"multus-44p7d\" (UID: \"e17d6c62-2952-420c-beda-49db831c48ac\") " pod="openshift-multus/multus-44p7d" Nov 03 01:17:26.572286 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.570341 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ecc2615-b458-41f8-839f-e22c53e7ad0e-metrics-certs\") pod \"network-metrics-daemon-8g56n\" (UID: \"6ecc2615-b458-41f8-839f-e22c53e7ad0e\") " pod="openshift-multus/network-metrics-daemon-8g56n" Nov 03 01:17:26.572286 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.570389 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7skt\" (UniqueName: \"kubernetes.io/projected/e17d6c62-2952-420c-beda-49db831c48ac-kube-api-access-g7skt\") pod \"multus-44p7d\" (UID: \"e17d6c62-2952-420c-beda-49db831c48ac\") " pod="openshift-multus/multus-44p7d" Nov 03 01:17:26.572286 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.570380 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e17d6c62-2952-420c-beda-49db831c48ac-system-cni-dir\") pod \"multus-44p7d\" (UID: \"e17d6c62-2952-420c-beda-49db831c48ac\") " pod="openshift-multus/multus-44p7d" Nov 03 01:17:26.572286 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.570417 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1febd8fa-5641-449b-b1fd-c7d02603fd1c-registration-dir\") pod \"aws-ebs-csi-driver-node-2jrfl\" (UID: \"1febd8fa-5641-449b-b1fd-c7d02603fd1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2jrfl" Nov 03 01:17:26.572286 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.570458 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d157814d-82d6-4af5-a648-c03ad35f3a41-etc-sysctl-conf\") pod \"tuned-c6r2x\" (UID: \"d157814d-82d6-4af5-a648-c03ad35f3a41\") " pod="openshift-cluster-node-tuning-operator/tuned-c6r2x" Nov 03 01:17:26.572286 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.570485 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d157814d-82d6-4af5-a648-c03ad35f3a41-lib-modules\") pod \"tuned-c6r2x\" (UID: \"d157814d-82d6-4af5-a648-c03ad35f3a41\") " pod="openshift-cluster-node-tuning-operator/tuned-c6r2x" Nov 03 01:17:26.572286 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.570486 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1febd8fa-5641-449b-b1fd-c7d02603fd1c-registration-dir\") pod \"aws-ebs-csi-driver-node-2jrfl\" (UID: \"1febd8fa-5641-449b-b1fd-c7d02603fd1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2jrfl" Nov 03 01:17:26.572286 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.570523 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fe434c5e-450b-4512-93aa-78d3e56e5002-konnectivity-ca\") pod \"konnectivity-agent-dd9jt\" (UID: \"fe434c5e-450b-4512-93aa-78d3e56e5002\") " pod="kube-system/konnectivity-agent-dd9jt" Nov 03 01:17:26.572286 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.570528 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e17d6c62-2952-420c-beda-49db831c48ac-host-run-multus-certs\") pod \"multus-44p7d\" (UID: \"e17d6c62-2952-420c-beda-49db831c48ac\") " pod="openshift-multus/multus-44p7d" Nov 03 01:17:26.572286 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:17:26.571228 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 01:17:26.572286 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.571130 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d157814d-82d6-4af5-a648-c03ad35f3a41-etc-sysctl-conf\") pod \"tuned-c6r2x\" (UID: \"d157814d-82d6-4af5-a648-c03ad35f3a41\") " pod="openshift-cluster-node-tuning-operator/tuned-c6r2x" Nov 03 01:17:26.572286 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.571257 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d157814d-82d6-4af5-a648-c03ad35f3a41-lib-modules\") pod \"tuned-c6r2x\" (UID: \"d157814d-82d6-4af5-a648-c03ad35f3a41\") " pod="openshift-cluster-node-tuning-operator/tuned-c6r2x" Nov 03 01:17:26.572286 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.570921 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4c25011b-c85c-4736-9f72-842a96949072-cni-binary-copy\") pod \"multus-additional-cni-plugins-jnvg6\" (UID: \"4c25011b-c85c-4736-9f72-842a96949072\") " pod="openshift-multus/multus-additional-cni-plugins-jnvg6" Nov 03 01:17:26.572286 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:17:26.571320 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ecc2615-b458-41f8-839f-e22c53e7ad0e-metrics-certs podName:6ecc2615-b458-41f8-839f-e22c53e7ad0e nodeName:}" failed. No retries permitted until 2026-11-03 01:17:27.071301152 +0000 UTC m=+3.158156713 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6ecc2615-b458-41f8-839f-e22c53e7ad0e-metrics-certs") pod "network-metrics-daemon-8g56n" (UID: "6ecc2615-b458-41f8-839f-e22c53e7ad0e") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 01:17:26.572286 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.571363 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e17d6c62-2952-420c-beda-49db831c48ac-cni-binary-copy\") pod \"multus-44p7d\" (UID: \"e17d6c62-2952-420c-beda-49db831c48ac\") " pod="openshift-multus/multus-44p7d" Nov 03 01:17:26.572286 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.571399 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e17d6c62-2952-420c-beda-49db831c48ac-multus-socket-dir-parent\") pod \"multus-44p7d\" (UID: \"e17d6c62-2952-420c-beda-49db831c48ac\") " pod="openshift-multus/multus-44p7d" Nov 03 01:17:26.573057 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.571519 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e17d6c62-2952-420c-beda-49db831c48ac-multus-socket-dir-parent\") pod \"multus-44p7d\" (UID: \"e17d6c62-2952-420c-beda-49db831c48ac\") " pod="openshift-multus/multus-44p7d" Nov 03 01:17:26.573057 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.571571 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d157814d-82d6-4af5-a648-c03ad35f3a41-etc-systemd\") pod \"tuned-c6r2x\" (UID: \"d157814d-82d6-4af5-a648-c03ad35f3a41\") " pod="openshift-cluster-node-tuning-operator/tuned-c6r2x" Nov 03 01:17:26.573057 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.571612 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d157814d-82d6-4af5-a648-c03ad35f3a41-etc-tuned\") pod \"tuned-c6r2x\" (UID: \"d157814d-82d6-4af5-a648-c03ad35f3a41\") " pod="openshift-cluster-node-tuning-operator/tuned-c6r2x" Nov 03 01:17:26.573057 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.571626 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzr84\" (UniqueName: \"kubernetes.io/projected/4c25011b-c85c-4736-9f72-842a96949072-kube-api-access-vzr84\") pod \"multus-additional-cni-plugins-jnvg6\" (UID: \"4c25011b-c85c-4736-9f72-842a96949072\") " pod="openshift-multus/multus-additional-cni-plugins-jnvg6" Nov 03 01:17:26.573057 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.571781 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d157814d-82d6-4af5-a648-c03ad35f3a41-etc-systemd\") pod \"tuned-c6r2x\" (UID: \"d157814d-82d6-4af5-a648-c03ad35f3a41\") " pod="openshift-cluster-node-tuning-operator/tuned-c6r2x" Nov 03 01:17:26.573057 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.571906 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e17d6c62-2952-420c-beda-49db831c48ac-multus-cni-dir\") pod \"multus-44p7d\" (UID: \"e17d6c62-2952-420c-beda-49db831c48ac\") " pod="openshift-multus/multus-44p7d" Nov 03 01:17:26.573057 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.571913 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4c25011b-c85c-4736-9f72-842a96949072-cni-binary-copy\") pod \"multus-additional-cni-plugins-jnvg6\" (UID: \"4c25011b-c85c-4736-9f72-842a96949072\") " pod="openshift-multus/multus-additional-cni-plugins-jnvg6" Nov 03 01:17:26.573057 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.571970 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e17d6c62-2952-420c-beda-49db831c48ac-cni-binary-copy\") pod \"multus-44p7d\" (UID: \"e17d6c62-2952-420c-beda-49db831c48ac\") " pod="openshift-multus/multus-44p7d" Nov 03 01:17:26.573057 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.571980 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88c5l\" (UniqueName: \"kubernetes.io/projected/bff481c2-d05e-4b45-827d-f5c8a0f60d0b-kube-api-access-88c5l\") pod \"node-ca-jpcqz\" (UID: \"bff481c2-d05e-4b45-827d-f5c8a0f60d0b\") " pod="openshift-image-registry/node-ca-jpcqz" Nov 03 01:17:26.573057 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.571990 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e17d6c62-2952-420c-beda-49db831c48ac-multus-cni-dir\") pod \"multus-44p7d\" (UID: \"e17d6c62-2952-420c-beda-49db831c48ac\") " pod="openshift-multus/multus-44p7d" Nov 03 01:17:26.573057 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.572049 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b56kh\" (UniqueName: \"kubernetes.io/projected/6ecc2615-b458-41f8-839f-e22c53e7ad0e-kube-api-access-b56kh\") pod \"network-metrics-daemon-8g56n\" (UID: \"6ecc2615-b458-41f8-839f-e22c53e7ad0e\") " pod="openshift-multus/network-metrics-daemon-8g56n" Nov 03 01:17:26.573057 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.572088 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d157814d-82d6-4af5-a648-c03ad35f3a41-sys\") pod \"tuned-c6r2x\" (UID: \"d157814d-82d6-4af5-a648-c03ad35f3a41\") " pod="openshift-cluster-node-tuning-operator/tuned-c6r2x" Nov 03 01:17:26.573057 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.572178 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d157814d-82d6-4af5-a648-c03ad35f3a41-sys\") pod \"tuned-c6r2x\" (UID: \"d157814d-82d6-4af5-a648-c03ad35f3a41\") " pod="openshift-cluster-node-tuning-operator/tuned-c6r2x" Nov 03 01:17:26.573057 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.572727 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d157814d-82d6-4af5-a648-c03ad35f3a41-tmp\") pod \"tuned-c6r2x\" (UID: \"d157814d-82d6-4af5-a648-c03ad35f3a41\") " pod="openshift-cluster-node-tuning-operator/tuned-c6r2x" Nov 03 01:17:26.575024 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.575003 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fe434c5e-450b-4512-93aa-78d3e56e5002-agent-certs\") pod \"konnectivity-agent-dd9jt\" (UID: \"fe434c5e-450b-4512-93aa-78d3e56e5002\") " pod="kube-system/konnectivity-agent-dd9jt" Nov 03 01:17:26.636016 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:17:26.635982 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 01:17:26.636016 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:17:26.636015 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 01:17:26.636172 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:17:26.636030 2578 projected.go:196] Error preparing data for projected volume kube-api-access-vgd8w for pod openshift-network-diagnostics/network-check-target-kg99x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 01:17:26.636172 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:17:26.636098 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3b2bf2f6-28aa-4480-9f4a-f4e7f659e966-kube-api-access-vgd8w podName:3b2bf2f6-28aa-4480-9f4a-f4e7f659e966 nodeName:}" failed. No retries permitted until 2026-11-03 01:17:27.13608185 +0000 UTC m=+3.222937431 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-vgd8w" (UniqueName: "kubernetes.io/projected/3b2bf2f6-28aa-4480-9f4a-f4e7f659e966-kube-api-access-vgd8w") pod "network-check-target-kg99x" (UID: "3b2bf2f6-28aa-4480-9f4a-f4e7f659e966") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 01:17:26.639104 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.639071 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2r8s\" (UniqueName: \"kubernetes.io/projected/20152fc3-0b5f-4c58-94fb-e02094377c74-kube-api-access-m2r8s\") pod \"iptables-alerter-x4tvh\" (UID: \"20152fc3-0b5f-4c58-94fb-e02094377c74\") " pod="openshift-network-operator/iptables-alerter-x4tvh" Nov 03 01:17:26.639104 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.639094 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cct5g\" (UniqueName: \"kubernetes.io/projected/d157814d-82d6-4af5-a648-c03ad35f3a41-kube-api-access-cct5g\") pod \"tuned-c6r2x\" (UID: \"d157814d-82d6-4af5-a648-c03ad35f3a41\") " pod="openshift-cluster-node-tuning-operator/tuned-c6r2x" Nov 03 01:17:26.639265 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.639145 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdm2b\" (UniqueName: \"kubernetes.io/projected/1febd8fa-5641-449b-b1fd-c7d02603fd1c-kube-api-access-bdm2b\") pod \"aws-ebs-csi-driver-node-2jrfl\" (UID: \"1febd8fa-5641-449b-b1fd-c7d02603fd1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2jrfl" Nov 03 01:17:26.639455 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.639434 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b56kh\" (UniqueName: \"kubernetes.io/projected/6ecc2615-b458-41f8-839f-e22c53e7ad0e-kube-api-access-b56kh\") pod \"network-metrics-daemon-8g56n\" (UID: \"6ecc2615-b458-41f8-839f-e22c53e7ad0e\") " pod="openshift-multus/network-metrics-daemon-8g56n" Nov 03 01:17:26.639526 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.639469 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7skt\" (UniqueName: \"kubernetes.io/projected/e17d6c62-2952-420c-beda-49db831c48ac-kube-api-access-g7skt\") pod \"multus-44p7d\" (UID: \"e17d6c62-2952-420c-beda-49db831c48ac\") " pod="openshift-multus/multus-44p7d" Nov 03 01:17:26.639526 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.639516 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzr84\" (UniqueName: \"kubernetes.io/projected/4c25011b-c85c-4736-9f72-842a96949072-kube-api-access-vzr84\") pod \"multus-additional-cni-plugins-jnvg6\" (UID: \"4c25011b-c85c-4736-9f72-842a96949072\") " pod="openshift-multus/multus-additional-cni-plugins-jnvg6" Nov 03 01:17:26.639599 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.639566 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88c5l\" (UniqueName: \"kubernetes.io/projected/bff481c2-d05e-4b45-827d-f5c8a0f60d0b-kube-api-access-88c5l\") pod \"node-ca-jpcqz\" (UID: \"bff481c2-d05e-4b45-827d-f5c8a0f60d0b\") " pod="openshift-image-registry/node-ca-jpcqz" Nov 03 01:17:26.660682 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.660660 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5qr9g" Nov 03 01:17:26.669944 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:26.669924 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacada406_fef9_4a03_ae21_c2351ef0a4cf.slice/crio-24bd13d85cff8f0cf46be8ea2d05895b41b5b5beed80366bec3a1469068ed1ab WatchSource:0}: Error finding container 24bd13d85cff8f0cf46be8ea2d05895b41b5b5beed80366bec3a1469068ed1ab: Status 404 returned error can't find the container with id 24bd13d85cff8f0cf46be8ea2d05895b41b5b5beed80366bec3a1469068ed1ab Nov 03 01:17:26.676564 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.676540 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-n2cjg" Nov 03 01:17:26.678241 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.678226 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-c6r2x" Nov 03 01:17:26.681935 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.681856 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-dd9jt" Nov 03 01:17:26.685281 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:26.685259 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabdd79d9_8d87_4133_85a2_643d26fc9faa.slice/crio-31db35db8133fd4c1b3f0909bcef183472dca46c17fb140c4240f9c733b82412 WatchSource:0}: Error finding container 31db35db8133fd4c1b3f0909bcef183472dca46c17fb140c4240f9c733b82412: Status 404 returned error can't find the container with id 31db35db8133fd4c1b3f0909bcef183472dca46c17fb140c4240f9c733b82412 Nov 03 01:17:26.687487 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:26.687466 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd157814d_82d6_4af5_a648_c03ad35f3a41.slice/crio-cdd70f502329ad7fb8bcf7318fb4a2b481469dcd239ef3f09f8148f315c04be9 WatchSource:0}: Error finding container cdd70f502329ad7fb8bcf7318fb4a2b481469dcd239ef3f09f8148f315c04be9: Status 404 returned error can't find the container with id cdd70f502329ad7fb8bcf7318fb4a2b481469dcd239ef3f09f8148f315c04be9 Nov 03 01:17:26.692039 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:26.692017 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe434c5e_450b_4512_93aa_78d3e56e5002.slice/crio-b47a157f9083e86acb094f9f6f448c186b115df3f282b88bae7bb034ef72ee4e WatchSource:0}: Error finding container b47a157f9083e86acb094f9f6f448c186b115df3f282b88bae7bb034ef72ee4e: Status 404 returned error can't find the container with id b47a157f9083e86acb094f9f6f448c186b115df3f282b88bae7bb034ef72ee4e Nov 03 01:17:26.692813 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.692798 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-jnvg6" Nov 03 01:17:26.696894 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.696878 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2jrfl" Nov 03 01:17:26.699616 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.699595 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-jpcqz" Nov 03 01:17:26.701728 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:26.701704 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c25011b_c85c_4736_9f72_842a96949072.slice/crio-3a98ca3c6966d3e44ce6dcd1ee13bd9cdf5b46bbeb0c34fe97865c015118ff10 WatchSource:0}: Error finding container 3a98ca3c6966d3e44ce6dcd1ee13bd9cdf5b46bbeb0c34fe97865c015118ff10: Status 404 returned error can't find the container with id 3a98ca3c6966d3e44ce6dcd1ee13bd9cdf5b46bbeb0c34fe97865c015118ff10 Nov 03 01:17:26.705402 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:26.705364 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1febd8fa_5641_449b_b1fd_c7d02603fd1c.slice/crio-406839857f2f8eaa16b794d2e388372be8131e40e4df221e5bea5891f1d3cd05 WatchSource:0}: Error finding container 406839857f2f8eaa16b794d2e388372be8131e40e4df221e5bea5891f1d3cd05: Status 404 returned error can't find the container with id 406839857f2f8eaa16b794d2e388372be8131e40e4df221e5bea5891f1d3cd05 Nov 03 01:17:26.706470 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.706450 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-44p7d" Nov 03 01:17:26.707616 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:26.707590 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbff481c2_d05e_4b45_827d_f5c8a0f60d0b.slice/crio-f1bbf393f14b3c0361a059eb428e4657c6a8c03af1b7fc3416c78ecb0a1423f1 WatchSource:0}: Error finding container f1bbf393f14b3c0361a059eb428e4657c6a8c03af1b7fc3416c78ecb0a1423f1: Status 404 returned error can't find the container with id f1bbf393f14b3c0361a059eb428e4657c6a8c03af1b7fc3416c78ecb0a1423f1 Nov 03 01:17:26.712531 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:26.712514 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-x4tvh" Nov 03 01:17:26.714963 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:26.714943 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode17d6c62_2952_420c_beda_49db831c48ac.slice/crio-ec11b93955fdbcf32dfeefb15d657d495d24cb563f8d5b07067303a2cc67c40a WatchSource:0}: Error finding container ec11b93955fdbcf32dfeefb15d657d495d24cb563f8d5b07067303a2cc67c40a: Status 404 returned error can't find the container with id ec11b93955fdbcf32dfeefb15d657d495d24cb563f8d5b07067303a2cc67c40a Nov 03 01:17:26.720692 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:17:26.720672 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20152fc3_0b5f_4c58_94fb_e02094377c74.slice/crio-24a6b7dfc849cfef3310febb8cba625fb2635841a98b3907ddfe9a3951c334ba WatchSource:0}: Error finding container 24a6b7dfc849cfef3310febb8cba625fb2635841a98b3907ddfe9a3951c334ba: Status 404 returned error can't find the container with id 24a6b7dfc849cfef3310febb8cba625fb2635841a98b3907ddfe9a3951c334ba Nov 03 01:17:27.075687 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:27.075603 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6179266f-07ff-48cd-bf03-3f1e961529d7-original-pull-secret\") pod \"global-pull-secret-syncer-dfr9t\" (UID: \"6179266f-07ff-48cd-bf03-3f1e961529d7\") " pod="kube-system/global-pull-secret-syncer-dfr9t" Nov 03 01:17:27.075687 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:27.075660 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ecc2615-b458-41f8-839f-e22c53e7ad0e-metrics-certs\") pod \"network-metrics-daemon-8g56n\" (UID: \"6ecc2615-b458-41f8-839f-e22c53e7ad0e\") " pod="openshift-multus/network-metrics-daemon-8g56n" Nov 03 01:17:27.075862 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:17:27.075751 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 01:17:27.075862 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:17:27.075817 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ecc2615-b458-41f8-839f-e22c53e7ad0e-metrics-certs podName:6ecc2615-b458-41f8-839f-e22c53e7ad0e nodeName:}" failed. No retries permitted until 2026-11-03 01:17:28.075802557 +0000 UTC m=+4.162658118 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6ecc2615-b458-41f8-839f-e22c53e7ad0e-metrics-certs") pod "network-metrics-daemon-8g56n" (UID: "6ecc2615-b458-41f8-839f-e22c53e7ad0e") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 01:17:27.075862 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:17:27.075753 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 01:17:27.075862 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:17:27.075851 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6179266f-07ff-48cd-bf03-3f1e961529d7-original-pull-secret podName:6179266f-07ff-48cd-bf03-3f1e961529d7 nodeName:}" failed. No retries permitted until 2026-11-03 01:17:28.075844512 +0000 UTC m=+4.162700073 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6179266f-07ff-48cd-bf03-3f1e961529d7-original-pull-secret") pod "global-pull-secret-syncer-dfr9t" (UID: "6179266f-07ff-48cd-bf03-3f1e961529d7") : object "kube-system"/"original-pull-secret" not registered Nov 03 01:17:27.176722 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:27.176682 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgd8w\" (UniqueName: \"kubernetes.io/projected/3b2bf2f6-28aa-4480-9f4a-f4e7f659e966-kube-api-access-vgd8w\") pod \"network-check-target-kg99x\" (UID: \"3b2bf2f6-28aa-4480-9f4a-f4e7f659e966\") " pod="openshift-network-diagnostics/network-check-target-kg99x" Nov 03 01:17:27.176904 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:17:27.176856 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 01:17:27.176904 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:17:27.176884 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 01:17:27.176904 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:17:27.176897 2578 projected.go:196] Error preparing data for projected volume kube-api-access-vgd8w for pod openshift-network-diagnostics/network-check-target-kg99x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 01:17:27.177057 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:17:27.176968 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3b2bf2f6-28aa-4480-9f4a-f4e7f659e966-kube-api-access-vgd8w podName:3b2bf2f6-28aa-4480-9f4a-f4e7f659e966 nodeName:}" failed. No retries permitted until 2026-11-03 01:17:28.176945209 +0000 UTC m=+4.263800789 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-vgd8w" (UniqueName: "kubernetes.io/projected/3b2bf2f6-28aa-4480-9f4a-f4e7f659e966-kube-api-access-vgd8w") pod "network-check-target-kg99x" (UID: "3b2bf2f6-28aa-4480-9f4a-f4e7f659e966") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 01:17:27.466589 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:27.466505 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 01:12:26 +0000 UTC" deadline="2028-07-28 03:11:58.993734668 +0000 UTC" Nov 03 01:17:27.466589 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:27.466535 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15193h54m31.527203081s" Nov 03 01:17:27.519118 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:27.519074 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-x4tvh" event={"ID":"20152fc3-0b5f-4c58-94fb-e02094377c74","Type":"ContainerStarted","Data":"24a6b7dfc849cfef3310febb8cba625fb2635841a98b3907ddfe9a3951c334ba"} Nov 03 01:17:27.520652 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:27.520622 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-44p7d" event={"ID":"e17d6c62-2952-420c-beda-49db831c48ac","Type":"ContainerStarted","Data":"ec11b93955fdbcf32dfeefb15d657d495d24cb563f8d5b07067303a2cc67c40a"} Nov 03 01:17:27.521744 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:27.521715 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jnvg6" event={"ID":"4c25011b-c85c-4736-9f72-842a96949072","Type":"ContainerStarted","Data":"3a98ca3c6966d3e44ce6dcd1ee13bd9cdf5b46bbeb0c34fe97865c015118ff10"} Nov 03 01:17:27.523130 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:27.523100 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5qr9g" event={"ID":"acada406-fef9-4a03-ae21-c2351ef0a4cf","Type":"ContainerStarted","Data":"24bd13d85cff8f0cf46be8ea2d05895b41b5b5beed80366bec3a1469068ed1ab"} Nov 03 01:17:27.525064 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:27.524470 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-jpcqz" event={"ID":"bff481c2-d05e-4b45-827d-f5c8a0f60d0b","Type":"ContainerStarted","Data":"f1bbf393f14b3c0361a059eb428e4657c6a8c03af1b7fc3416c78ecb0a1423f1"} Nov 03 01:17:27.525695 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:27.525670 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2jrfl" event={"ID":"1febd8fa-5641-449b-b1fd-c7d02603fd1c","Type":"ContainerStarted","Data":"406839857f2f8eaa16b794d2e388372be8131e40e4df221e5bea5891f1d3cd05"} Nov 03 01:17:27.527010 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:27.526986 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-dd9jt" event={"ID":"fe434c5e-450b-4512-93aa-78d3e56e5002","Type":"ContainerStarted","Data":"b47a157f9083e86acb094f9f6f448c186b115df3f282b88bae7bb034ef72ee4e"} Nov 03 01:17:27.528505 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:27.528483 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-c6r2x" event={"ID":"d157814d-82d6-4af5-a648-c03ad35f3a41","Type":"ContainerStarted","Data":"cdd70f502329ad7fb8bcf7318fb4a2b481469dcd239ef3f09f8148f315c04be9"} Nov 03 01:17:27.529729 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:27.529707 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n2cjg" event={"ID":"abdd79d9-8d87-4133-85a2-643d26fc9faa","Type":"ContainerStarted","Data":"31db35db8133fd4c1b3f0909bcef183472dca46c17fb140c4240f9c733b82412"} Nov 03 01:17:28.085925 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:28.085826 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ecc2615-b458-41f8-839f-e22c53e7ad0e-metrics-certs\") pod \"network-metrics-daemon-8g56n\" (UID: \"6ecc2615-b458-41f8-839f-e22c53e7ad0e\") " pod="openshift-multus/network-metrics-daemon-8g56n" Nov 03 01:17:28.086126 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:28.086026 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6179266f-07ff-48cd-bf03-3f1e961529d7-original-pull-secret\") pod \"global-pull-secret-syncer-dfr9t\" (UID: \"6179266f-07ff-48cd-bf03-3f1e961529d7\") " pod="kube-system/global-pull-secret-syncer-dfr9t" Nov 03 01:17:28.086233 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:17:28.086124 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 01:17:28.086233 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:17:28.086190 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6179266f-07ff-48cd-bf03-3f1e961529d7-original-pull-secret podName:6179266f-07ff-48cd-bf03-3f1e961529d7 nodeName:}" failed. No retries permitted until 2026-11-03 01:17:30.08617101 +0000 UTC m=+6.173026586 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6179266f-07ff-48cd-bf03-3f1e961529d7-original-pull-secret") pod "global-pull-secret-syncer-dfr9t" (UID: "6179266f-07ff-48cd-bf03-3f1e961529d7") : object "kube-system"/"original-pull-secret" not registered Nov 03 01:17:28.086233 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:17:28.086212 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 01:17:28.086527 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:17:28.086243 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ecc2615-b458-41f8-839f-e22c53e7ad0e-metrics-certs podName:6ecc2615-b458-41f8-839f-e22c53e7ad0e nodeName:}" failed. No retries permitted until 2026-11-03 01:17:30.086233082 +0000 UTC m=+6.173088662 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6ecc2615-b458-41f8-839f-e22c53e7ad0e-metrics-certs") pod "network-metrics-daemon-8g56n" (UID: "6ecc2615-b458-41f8-839f-e22c53e7ad0e") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 01:17:28.187272 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:28.187224 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgd8w\" (UniqueName: \"kubernetes.io/projected/3b2bf2f6-28aa-4480-9f4a-f4e7f659e966-kube-api-access-vgd8w\") pod \"network-check-target-kg99x\" (UID: \"3b2bf2f6-28aa-4480-9f4a-f4e7f659e966\") " pod="openshift-network-diagnostics/network-check-target-kg99x" Nov 03 01:17:28.187503 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:17:28.187460 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 01:17:28.187503 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:17:28.187481 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 01:17:28.187597 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:17:28.187526 2578 projected.go:196] Error preparing data for projected volume kube-api-access-vgd8w for pod openshift-network-diagnostics/network-check-target-kg99x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 01:17:28.187667 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:17:28.187595 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3b2bf2f6-28aa-4480-9f4a-f4e7f659e966-kube-api-access-vgd8w podName:3b2bf2f6-28aa-4480-9f4a-f4e7f659e966 nodeName:}" failed. No retries permitted until 2026-11-03 01:17:30.1875742 +0000 UTC m=+6.274429780 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-vgd8w" (UniqueName: "kubernetes.io/projected/3b2bf2f6-28aa-4480-9f4a-f4e7f659e966-kube-api-access-vgd8w") pod "network-check-target-kg99x" (UID: "3b2bf2f6-28aa-4480-9f4a-f4e7f659e966") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 01:17:28.467141 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:28.467025 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 01:12:26 +0000 UTC" deadline="2028-04-10 15:46:02.998556645 +0000 UTC" Nov 03 01:17:28.467141 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:28.467090 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12590h28m34.531471505s" Nov 03 01:17:28.512975 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:28.512938 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8g56n" Nov 03 01:17:28.513151 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:17:28.513124 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8g56n" podUID="6ecc2615-b458-41f8-839f-e22c53e7ad0e" Nov 03 01:17:28.513237 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:28.513220 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kg99x" Nov 03 01:17:28.513322 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:17:28.513302 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kg99x" podUID="3b2bf2f6-28aa-4480-9f4a-f4e7f659e966" Nov 03 01:17:28.513401 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:28.513386 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dfr9t" Nov 03 01:17:28.513530 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:17:28.513508 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dfr9t" podUID="6179266f-07ff-48cd-bf03-3f1e961529d7" Nov 03 01:17:30.109161 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:30.108803 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6179266f-07ff-48cd-bf03-3f1e961529d7-original-pull-secret\") pod \"global-pull-secret-syncer-dfr9t\" (UID: \"6179266f-07ff-48cd-bf03-3f1e961529d7\") " pod="kube-system/global-pull-secret-syncer-dfr9t" Nov 03 01:17:30.109161 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:30.108889 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ecc2615-b458-41f8-839f-e22c53e7ad0e-metrics-certs\") pod \"network-metrics-daemon-8g56n\" (UID: \"6ecc2615-b458-41f8-839f-e22c53e7ad0e\") " pod="openshift-multus/network-metrics-daemon-8g56n" Nov 03 01:17:30.109161 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:17:30.108999 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 01:17:30.109161 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:17:30.109029 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 01:17:30.109161 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:17:30.109080 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6179266f-07ff-48cd-bf03-3f1e961529d7-original-pull-secret podName:6179266f-07ff-48cd-bf03-3f1e961529d7 nodeName:}" failed. No retries permitted until 2026-11-03 01:17:34.109058174 +0000 UTC m=+10.195913735 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6179266f-07ff-48cd-bf03-3f1e961529d7-original-pull-secret") pod "global-pull-secret-syncer-dfr9t" (UID: "6179266f-07ff-48cd-bf03-3f1e961529d7") : object "kube-system"/"original-pull-secret" not registered Nov 03 01:17:30.109161 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:17:30.109098 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ecc2615-b458-41f8-839f-e22c53e7ad0e-metrics-certs podName:6ecc2615-b458-41f8-839f-e22c53e7ad0e nodeName:}" failed. No retries permitted until 2026-11-03 01:17:34.109090751 +0000 UTC m=+10.195946313 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6ecc2615-b458-41f8-839f-e22c53e7ad0e-metrics-certs") pod "network-metrics-daemon-8g56n" (UID: "6ecc2615-b458-41f8-839f-e22c53e7ad0e") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 01:17:30.211783 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:30.211746 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgd8w\" (UniqueName: \"kubernetes.io/projected/3b2bf2f6-28aa-4480-9f4a-f4e7f659e966-kube-api-access-vgd8w\") pod \"network-check-target-kg99x\" (UID: \"3b2bf2f6-28aa-4480-9f4a-f4e7f659e966\") " pod="openshift-network-diagnostics/network-check-target-kg99x" Nov 03 01:17:30.211983 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:17:30.211965 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 01:17:30.212119 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:17:30.211992 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 01:17:30.212119 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:17:30.212005 2578 projected.go:196] Error preparing data for projected volume kube-api-access-vgd8w for pod openshift-network-diagnostics/network-check-target-kg99x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 01:17:30.212119 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:17:30.212077 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3b2bf2f6-28aa-4480-9f4a-f4e7f659e966-kube-api-access-vgd8w podName:3b2bf2f6-28aa-4480-9f4a-f4e7f659e966 nodeName:}" failed. No retries permitted until 2026-11-03 01:17:34.212055442 +0000 UTC m=+10.298911006 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-vgd8w" (UniqueName: "kubernetes.io/projected/3b2bf2f6-28aa-4480-9f4a-f4e7f659e966-kube-api-access-vgd8w") pod "network-check-target-kg99x" (UID: "3b2bf2f6-28aa-4480-9f4a-f4e7f659e966") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 01:17:30.513184 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:30.512937 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kg99x" Nov 03 01:17:30.513321 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:17:30.513289 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kg99x" podUID="3b2bf2f6-28aa-4480-9f4a-f4e7f659e966" Nov 03 01:17:30.514057 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:30.514037 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dfr9t" Nov 03 01:17:30.514185 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:17:30.514159 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dfr9t" podUID="6179266f-07ff-48cd-bf03-3f1e961529d7" Nov 03 01:17:30.514242 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:30.514229 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8g56n" Nov 03 01:17:30.514340 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:17:30.514321 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8g56n" podUID="6ecc2615-b458-41f8-839f-e22c53e7ad0e" Nov 03 01:17:31.563215 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:31.563168 2578 generic.go:356] "Generic (PLEG): container finished" podID="f7e070a0c923c611d3cf96ab59dc8b57" containerID="6fb6612237ca18debdc4ee17b23d29d97c5806aea283b7663f3d2636bc452fbf" exitCode=0 Nov 03 01:17:31.563738 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:31.563272 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-16.us-west-2.compute.internal" event={"ID":"f7e070a0c923c611d3cf96ab59dc8b57","Type":"ContainerDied","Data":"6fb6612237ca18debdc4ee17b23d29d97c5806aea283b7663f3d2636bc452fbf"} Nov 03 01:17:31.579303 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:31.578646 2578 generic.go:356] "Generic (PLEG): container finished" podID="4c25011b-c85c-4736-9f72-842a96949072" containerID="6ff76455930533288223388e53328c54a4250811df4749a54a19a17c3c177f6f" exitCode=0 Nov 03 01:17:31.579303 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:31.578698 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jnvg6" event={"ID":"4c25011b-c85c-4736-9f72-842a96949072","Type":"ContainerDied","Data":"6ff76455930533288223388e53328c54a4250811df4749a54a19a17c3c177f6f"} Nov 03 01:17:32.510275 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:32.510197 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dfr9t" Nov 03 01:17:32.510492 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:17:32.510310 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dfr9t" podUID="6179266f-07ff-48cd-bf03-3f1e961529d7" Nov 03 01:17:32.510492 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:32.510369 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8g56n" Nov 03 01:17:32.510492 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:17:32.510477 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8g56n" podUID="6ecc2615-b458-41f8-839f-e22c53e7ad0e" Nov 03 01:17:32.510696 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:32.510530 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kg99x" Nov 03 01:17:32.510696 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:17:32.510595 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kg99x" podUID="3b2bf2f6-28aa-4480-9f4a-f4e7f659e966" Nov 03 01:17:32.587185 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:32.587144 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-16.us-west-2.compute.internal" event={"ID":"f7e070a0c923c611d3cf96ab59dc8b57","Type":"ContainerStarted","Data":"3bad95a8bbb41660529b2d2ea4631a6b90124d9ab2f6ad4cef3327f64d303298"} Nov 03 01:17:32.591473 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:32.591442 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-16.us-west-2.compute.internal" event={"ID":"7f364a286e3c240d587086e0ec2180aa","Type":"ContainerStarted","Data":"b235ef66737c2cad06a4a918d29bae2112b4568db459177d9e5e853b83cf8792"} Nov 03 01:17:32.595928 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:32.595896 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-dd9jt" event={"ID":"fe434c5e-450b-4512-93aa-78d3e56e5002","Type":"ContainerStarted","Data":"823a5ad3e941f0ed98d315fb037ad452e2c21c13f2551770644c02e754333205"} Nov 03 01:17:32.716288 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:32.716227 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-16.us-west-2.compute.internal" podStartSLOduration=6.716210087 podStartE2EDuration="6.716210087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 01:17:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:17:32.715641675 +0000 UTC m=+8.802497264" watchObservedRunningTime="2026-11-03 01:17:32.716210087 +0000 UTC m=+8.803065674" Nov 03 01:17:32.981411 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:32.981335 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-16.us-west-2.compute.internal" podStartSLOduration=6.981310698 podStartE2EDuration="6.981310698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 01:17:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:17:32.980862122 +0000 UTC m=+9.067717708" watchObservedRunningTime="2026-11-03 01:17:32.981310698 +0000 UTC m=+9.068166287" Nov 03 01:17:32.981712 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:32.981665 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-dd9jt" podStartSLOduration=2.516894125 podStartE2EDuration="7.981654068s" totalImagesPullingTime="5.464759943s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:17:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:17:32.848225354 +0000 UTC m=+8.935080941" watchObservedRunningTime="2026-11-03 01:17:32.981654068 +0000 UTC m=+9.068509653" Nov 03 01:17:33.603712 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:33.601488 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-jpcqz" event={"ID":"bff481c2-d05e-4b45-827d-f5c8a0f60d0b","Type":"ContainerStarted","Data":"faf927e4b1f782bb2105e042637707b11e88d532d3261c467f3306b7a15edded"} Nov 03 01:17:33.730457 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:33.730376 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-jpcqz" podStartSLOduration=2.158703771 podStartE2EDuration="8.73035879s" totalImagesPullingTime="6.571655019s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 01:17:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:17:33.729642808 +0000 UTC m=+9.816498394" watchObservedRunningTime="2026-11-03 01:17:33.73035879 +0000 UTC m=+9.817214376" Nov 03 01:17:33.970434 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:33.970334 2578 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-dd9jt" Nov 03 01:17:33.971036 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:33.971014 2578 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-dd9jt" Nov 03 01:17:34.142487 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:34.142442 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ecc2615-b458-41f8-839f-e22c53e7ad0e-metrics-certs\") pod \"network-metrics-daemon-8g56n\" (UID: \"6ecc2615-b458-41f8-839f-e22c53e7ad0e\") " pod="openshift-multus/network-metrics-daemon-8g56n" Nov 03 01:17:34.142640 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:34.142509 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6179266f-07ff-48cd-bf03-3f1e961529d7-original-pull-secret\") pod \"global-pull-secret-syncer-dfr9t\" (UID: \"6179266f-07ff-48cd-bf03-3f1e961529d7\") " pod="kube-system/global-pull-secret-syncer-dfr9t" Nov 03 01:17:34.142640 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:17:34.142631 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 01:17:34.142739 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:17:34.142683 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6179266f-07ff-48cd-bf03-3f1e961529d7-original-pull-secret podName:6179266f-07ff-48cd-bf03-3f1e961529d7 nodeName:}" failed. No retries permitted until 2026-11-03 01:17:42.142669785 +0000 UTC m=+18.229525346 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6179266f-07ff-48cd-bf03-3f1e961529d7-original-pull-secret") pod "global-pull-secret-syncer-dfr9t" (UID: "6179266f-07ff-48cd-bf03-3f1e961529d7") : object "kube-system"/"original-pull-secret" not registered Nov 03 01:17:34.143093 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:17:34.143072 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 01:17:34.143162 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:17:34.143135 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ecc2615-b458-41f8-839f-e22c53e7ad0e-metrics-certs podName:6ecc2615-b458-41f8-839f-e22c53e7ad0e nodeName:}" failed. No retries permitted until 2026-11-03 01:17:42.143117459 +0000 UTC m=+18.229973036 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6ecc2615-b458-41f8-839f-e22c53e7ad0e-metrics-certs") pod "network-metrics-daemon-8g56n" (UID: "6ecc2615-b458-41f8-839f-e22c53e7ad0e") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 01:17:34.245323 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:34.244790 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgd8w\" (UniqueName: \"kubernetes.io/projected/3b2bf2f6-28aa-4480-9f4a-f4e7f659e966-kube-api-access-vgd8w\") pod \"network-check-target-kg99x\" (UID: \"3b2bf2f6-28aa-4480-9f4a-f4e7f659e966\") " pod="openshift-network-diagnostics/network-check-target-kg99x" Nov 03 01:17:34.245323 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:17:34.244963 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 01:17:34.245323 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:17:34.244986 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 01:17:34.245323 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:17:34.245000 2578 projected.go:196] Error preparing data for projected volume kube-api-access-vgd8w for pod openshift-network-diagnostics/network-check-target-kg99x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 01:17:34.245323 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:17:34.245065 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3b2bf2f6-28aa-4480-9f4a-f4e7f659e966-kube-api-access-vgd8w podName:3b2bf2f6-28aa-4480-9f4a-f4e7f659e966 nodeName:}" failed. No retries permitted until 2026-11-03 01:17:42.245046214 +0000 UTC m=+18.331901790 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-vgd8w" (UniqueName: "kubernetes.io/projected/3b2bf2f6-28aa-4480-9f4a-f4e7f659e966-kube-api-access-vgd8w") pod "network-check-target-kg99x" (UID: "3b2bf2f6-28aa-4480-9f4a-f4e7f659e966") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 01:17:34.510750 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:34.510664 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dfr9t" Nov 03 01:17:34.510972 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:17:34.510789 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dfr9t" podUID="6179266f-07ff-48cd-bf03-3f1e961529d7" Nov 03 01:17:34.510972 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:34.510869 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8g56n" Nov 03 01:17:34.510972 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:17:34.510964 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8g56n" podUID="6ecc2615-b458-41f8-839f-e22c53e7ad0e" Nov 03 01:17:34.511141 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:34.511021 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kg99x" Nov 03 01:17:34.511141 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:17:34.511088 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kg99x" podUID="3b2bf2f6-28aa-4480-9f4a-f4e7f659e966" Nov 03 01:17:34.617724 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:34.617114 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2jrfl" event={"ID":"1febd8fa-5641-449b-b1fd-c7d02603fd1c","Type":"ContainerStarted","Data":"9270fcda861a803c611f6c72298a522481b65bb36202fa3ba365af843f1803cb"} Nov 03 01:17:34.617724 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:34.617179 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-dd9jt" Nov 03 01:17:34.618597 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:34.618414 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-dd9jt" Nov 03 01:17:35.044101 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:35.044062 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:35.303560 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:35.303439 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:36.283037 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:36.282889 2578 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 03 01:17:36.456745 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:36.456648 2578 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T01:17:36.283034765Z","UUID":"406777a4-9360-4c34-bbd0-559aedc7f801","Handler":null,"Name":"","Endpoint":""} Nov 03 01:17:36.459717 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:36.459695 2578 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 03 01:17:36.459717 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:36.459721 2578 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 03 01:17:36.513910 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:36.513821 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8g56n" Nov 03 01:17:36.513910 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:36.513901 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dfr9t" Nov 03 01:17:36.514077 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:36.513985 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kg99x" Nov 03 01:17:36.514077 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:17:36.513999 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8g56n" podUID="6ecc2615-b458-41f8-839f-e22c53e7ad0e" Nov 03 01:17:36.514134 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:17:36.514075 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kg99x" podUID="3b2bf2f6-28aa-4480-9f4a-f4e7f659e966" Nov 03 01:17:36.514175 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:17:36.514141 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dfr9t" podUID="6179266f-07ff-48cd-bf03-3f1e961529d7" Nov 03 01:17:36.623816 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:36.623771 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jnvg6" event={"ID":"4c25011b-c85c-4736-9f72-842a96949072","Type":"ContainerStarted","Data":"d9ae2a77a51cd08fc6eafd5480b933293ed28b4b72b4207ec38186ac55d18245"} Nov 03 01:17:36.625759 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:36.625727 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5qr9g" event={"ID":"acada406-fef9-4a03-ae21-c2351ef0a4cf","Type":"ContainerStarted","Data":"449081cb5b8f7eef5baa3a49762927d4fdaf87c32a02da7fc665831e228ca814"} Nov 03 01:17:36.628238 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:36.628208 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2jrfl" event={"ID":"1febd8fa-5641-449b-b1fd-c7d02603fd1c","Type":"ContainerStarted","Data":"e51f01c2157e51180deed0401f92c039870cb77bd7352e6bd7ccffda19a4fe02"} Nov 03 01:17:36.630450 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:36.630413 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-44p7d" event={"ID":"e17d6c62-2952-420c-beda-49db831c48ac","Type":"ContainerStarted","Data":"4ba6f80ef01650d23db2a61911de1cf2ae83c142283883ecbdba992065194b8b"} Nov 03 01:17:36.883875 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:36.883813 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-44p7d" podStartSLOduration=2.95055096 podStartE2EDuration="11.883792458s" totalImagesPullingTime="8.933241498s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 01:17:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:17:36.883641487 +0000 UTC m=+12.970497095" watchObservedRunningTime="2026-11-03 01:17:36.883792458 +0000 UTC m=+12.970648042" Nov 03 01:17:37.001846 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:37.001783 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-5qr9g" podStartSLOduration=2.519307713 podStartE2EDuration="12.001768764s" totalImagesPullingTime="9.482461051s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 01:17:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:17:37.001208204 +0000 UTC m=+13.088063812" watchObservedRunningTime="2026-11-03 01:17:37.001768764 +0000 UTC m=+13.088624346" Nov 03 01:17:37.633893 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:37.633814 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-x4tvh" event={"ID":"20152fc3-0b5f-4c58-94fb-e02094377c74","Type":"ContainerStarted","Data":"e40b0a22ee3b64eae628bd9bb9c8fe386c175401133c5f1370e78dee5b92a4c2"} Nov 03 01:17:37.635504 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:37.635481 2578 generic.go:356] "Generic (PLEG): container finished" podID="4c25011b-c85c-4736-9f72-842a96949072" containerID="d9ae2a77a51cd08fc6eafd5480b933293ed28b4b72b4207ec38186ac55d18245" exitCode=0 Nov 03 01:17:37.635582 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:37.635562 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jnvg6" event={"ID":"4c25011b-c85c-4736-9f72-842a96949072","Type":"ContainerDied","Data":"d9ae2a77a51cd08fc6eafd5480b933293ed28b4b72b4207ec38186ac55d18245"} Nov 03 01:17:37.884878 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:37.884785 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-x4tvh" podStartSLOduration=3.681178226 podStartE2EDuration="12.884769875s" totalImagesPullingTime="9.203591649s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 01:17:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:17:37.759866178 +0000 UTC m=+13.846721760" watchObservedRunningTime="2026-11-03 01:17:37.884769875 +0000 UTC m=+13.971625455" Nov 03 01:17:38.510982 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:38.510950 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dfr9t" Nov 03 01:17:38.510982 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:38.510979 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kg99x" Nov 03 01:17:38.511226 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:17:38.511110 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dfr9t" podUID="6179266f-07ff-48cd-bf03-3f1e961529d7" Nov 03 01:17:38.511226 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:38.511154 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8g56n" Nov 03 01:17:38.511226 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:17:38.511216 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kg99x" podUID="3b2bf2f6-28aa-4480-9f4a-f4e7f659e966" Nov 03 01:17:38.511353 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:17:38.511279 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8g56n" podUID="6ecc2615-b458-41f8-839f-e22c53e7ad0e" Nov 03 01:17:38.638995 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:38.638964 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2jrfl" event={"ID":"1febd8fa-5641-449b-b1fd-c7d02603fd1c","Type":"ContainerStarted","Data":"8cd41a89fa1f69927ceb4a86e491ebcd2ab1a94b9d175b60fafb45e4e0f4ecdd"} Nov 03 01:17:38.640559 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:38.640535 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-c6r2x" event={"ID":"d157814d-82d6-4af5-a648-c03ad35f3a41","Type":"ContainerStarted","Data":"41b1772591577ff5e7153b6be2c64d6f05c2e243cd295b1640478a77168710c0"} Nov 03 01:17:38.760042 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:38.759985 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2jrfl" podStartSLOduration=2.811060708 podStartE2EDuration="13.759965964s" totalImagesPullingTime="10.948905256s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 01:17:25 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:17:38.759927192 +0000 UTC m=+14.846782777" watchObservedRunningTime="2026-11-03 01:17:38.759965964 +0000 UTC m=+14.846821541" Nov 03 01:17:38.884238 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:38.884173 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-c6r2x" podStartSLOduration=2.673515043 podStartE2EDuration="13.884154333s" totalImagesPullingTime="11.21063929s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 01:17:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:17:38.88394009 +0000 UTC m=+14.970795672" watchObservedRunningTime="2026-11-03 01:17:38.884154333 +0000 UTC m=+14.971009916" Nov 03 01:17:39.586595 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:39.586555 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:39.645075 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:39.644886 2578 generic.go:356] "Generic (PLEG): container finished" podID="4c25011b-c85c-4736-9f72-842a96949072" containerID="10d60ccdb372fc5c82899df0ff62f14f2c97e4a68cee4802f08bf91d025ae9e8" exitCode=0 Nov 03 01:17:39.645505 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:39.644967 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jnvg6" event={"ID":"4c25011b-c85c-4736-9f72-842a96949072","Type":"ContainerDied","Data":"10d60ccdb372fc5c82899df0ff62f14f2c97e4a68cee4802f08bf91d025ae9e8"} Nov 03 01:17:39.781725 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:39.781693 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:40.511005 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:40.510972 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kg99x" Nov 03 01:17:40.511005 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:40.511020 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8g56n" Nov 03 01:17:40.511005 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:40.511039 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dfr9t" Nov 03 01:17:40.511451 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:17:40.511178 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8g56n" podUID="6ecc2615-b458-41f8-839f-e22c53e7ad0e" Nov 03 01:17:40.511451 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:17:40.511247 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kg99x" podUID="3b2bf2f6-28aa-4480-9f4a-f4e7f659e966" Nov 03 01:17:40.511451 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:17:40.511316 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dfr9t" podUID="6179266f-07ff-48cd-bf03-3f1e961529d7" Nov 03 01:17:41.651672 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:41.651637 2578 generic.go:356] "Generic (PLEG): container finished" podID="4c25011b-c85c-4736-9f72-842a96949072" containerID="f2d3817df2b9d6350149ec0853e36b2981b85e189e92d129dd414d56096fa1c9" exitCode=0 Nov 03 01:17:41.652130 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:41.651693 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jnvg6" event={"ID":"4c25011b-c85c-4736-9f72-842a96949072","Type":"ContainerDied","Data":"f2d3817df2b9d6350149ec0853e36b2981b85e189e92d129dd414d56096fa1c9"} Nov 03 01:17:41.698868 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:41.698836 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:41.917447 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:41.917354 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:42.121147 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:42.121116 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:42.203479 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:42.203395 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6179266f-07ff-48cd-bf03-3f1e961529d7-original-pull-secret\") pod \"global-pull-secret-syncer-dfr9t\" (UID: \"6179266f-07ff-48cd-bf03-3f1e961529d7\") " pod="kube-system/global-pull-secret-syncer-dfr9t" Nov 03 01:17:42.203479 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:42.203452 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ecc2615-b458-41f8-839f-e22c53e7ad0e-metrics-certs\") pod \"network-metrics-daemon-8g56n\" (UID: \"6ecc2615-b458-41f8-839f-e22c53e7ad0e\") " pod="openshift-multus/network-metrics-daemon-8g56n" Nov 03 01:17:42.203657 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:17:42.203536 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 01:17:42.203657 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:17:42.203538 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 01:17:42.203657 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:17:42.203590 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ecc2615-b458-41f8-839f-e22c53e7ad0e-metrics-certs podName:6ecc2615-b458-41f8-839f-e22c53e7ad0e nodeName:}" failed. No retries permitted until 2026-11-03 01:17:58.203574895 +0000 UTC m=+34.290430454 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6ecc2615-b458-41f8-839f-e22c53e7ad0e-metrics-certs") pod "network-metrics-daemon-8g56n" (UID: "6ecc2615-b458-41f8-839f-e22c53e7ad0e") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 01:17:42.203657 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:17:42.203604 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6179266f-07ff-48cd-bf03-3f1e961529d7-original-pull-secret podName:6179266f-07ff-48cd-bf03-3f1e961529d7 nodeName:}" failed. No retries permitted until 2026-11-03 01:17:58.203597762 +0000 UTC m=+34.290453321 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6179266f-07ff-48cd-bf03-3f1e961529d7-original-pull-secret") pod "global-pull-secret-syncer-dfr9t" (UID: "6179266f-07ff-48cd-bf03-3f1e961529d7") : object "kube-system"/"original-pull-secret" not registered Nov 03 01:17:42.304521 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:42.304491 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgd8w\" (UniqueName: \"kubernetes.io/projected/3b2bf2f6-28aa-4480-9f4a-f4e7f659e966-kube-api-access-vgd8w\") pod \"network-check-target-kg99x\" (UID: \"3b2bf2f6-28aa-4480-9f4a-f4e7f659e966\") " pod="openshift-network-diagnostics/network-check-target-kg99x" Nov 03 01:17:42.304681 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:17:42.304614 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 01:17:42.304681 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:17:42.304627 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 01:17:42.304681 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:17:42.304636 2578 projected.go:196] Error preparing data for projected volume kube-api-access-vgd8w for pod openshift-network-diagnostics/network-check-target-kg99x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 01:17:42.304681 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:17:42.304680 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3b2bf2f6-28aa-4480-9f4a-f4e7f659e966-kube-api-access-vgd8w podName:3b2bf2f6-28aa-4480-9f4a-f4e7f659e966 nodeName:}" failed. No retries permitted until 2026-11-03 01:17:58.304669225 +0000 UTC m=+34.391524784 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-vgd8w" (UniqueName: "kubernetes.io/projected/3b2bf2f6-28aa-4480-9f4a-f4e7f659e966-kube-api-access-vgd8w") pod "network-check-target-kg99x" (UID: "3b2bf2f6-28aa-4480-9f4a-f4e7f659e966") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 01:17:42.326693 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:42.326662 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:42.510950 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:42.510870 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kg99x" Nov 03 01:17:42.511108 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:42.510869 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8g56n" Nov 03 01:17:42.511108 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:17:42.510981 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kg99x" podUID="3b2bf2f6-28aa-4480-9f4a-f4e7f659e966" Nov 03 01:17:42.511108 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:17:42.511081 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8g56n" podUID="6ecc2615-b458-41f8-839f-e22c53e7ad0e" Nov 03 01:17:42.511108 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:42.510879 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dfr9t" Nov 03 01:17:42.511318 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:17:42.511180 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dfr9t" podUID="6179266f-07ff-48cd-bf03-3f1e961529d7" Nov 03 01:17:42.528681 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:42.528631 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:42.950067 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:42.950036 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:43.149127 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:43.149074 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:44.339563 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:44.339521 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:44.511720 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:44.511681 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dfr9t" Nov 03 01:17:44.511903 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:44.511779 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8g56n" Nov 03 01:17:44.511903 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:44.511823 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kg99x" Nov 03 01:17:44.511903 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:17:44.511828 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dfr9t" podUID="6179266f-07ff-48cd-bf03-3f1e961529d7" Nov 03 01:17:44.512024 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:17:44.511896 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kg99x" podUID="3b2bf2f6-28aa-4480-9f4a-f4e7f659e966" Nov 03 01:17:44.512024 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:17:44.511978 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8g56n" podUID="6ecc2615-b458-41f8-839f-e22c53e7ad0e" Nov 03 01:17:44.543026 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:44.542995 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:44.662053 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:44.661971 2578 generic.go:356] "Generic (PLEG): container finished" podID="4c25011b-c85c-4736-9f72-842a96949072" containerID="6cdb7dd6dda109a84fc562fdbbdb82f57b1618c0f25acc2e5f4ee17d4257a65b" exitCode=0 Nov 03 01:17:44.662053 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:44.662028 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jnvg6" event={"ID":"4c25011b-c85c-4736-9f72-842a96949072","Type":"ContainerDied","Data":"6cdb7dd6dda109a84fc562fdbbdb82f57b1618c0f25acc2e5f4ee17d4257a65b"} Nov 03 01:17:45.305266 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:45.305219 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:45.667886 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:45.667808 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:45.668317 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:45.667934 2578 generic.go:356] "Generic (PLEG): container finished" podID="4c25011b-c85c-4736-9f72-842a96949072" containerID="707a60ffd5841e43a67ccd194a43baf64a156b90613edd5247d050faf9033b68" exitCode=0 Nov 03 01:17:45.668317 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:45.667985 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jnvg6" event={"ID":"4c25011b-c85c-4736-9f72-842a96949072","Type":"ContainerDied","Data":"707a60ffd5841e43a67ccd194a43baf64a156b90613edd5247d050faf9033b68"} Nov 03 01:17:46.188862 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:46.188835 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5qr9g_acada406-fef9-4a03-ae21-c2351ef0a4cf/dns-node-resolver/0.log" Nov 03 01:17:46.371134 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:46.371099 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:46.511189 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:46.511117 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kg99x" Nov 03 01:17:46.511189 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:46.511132 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8g56n" Nov 03 01:17:46.511402 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:17:46.511234 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kg99x" podUID="3b2bf2f6-28aa-4480-9f4a-f4e7f659e966" Nov 03 01:17:46.511402 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:46.511335 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dfr9t" Nov 03 01:17:46.511553 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:17:46.511521 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8g56n" podUID="6ecc2615-b458-41f8-839f-e22c53e7ad0e" Nov 03 01:17:46.511686 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:17:46.511655 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dfr9t" podUID="6179266f-07ff-48cd-bf03-3f1e961529d7" Nov 03 01:17:46.672922 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:46.672676 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n2cjg" event={"ID":"abdd79d9-8d87-4133-85a2-643d26fc9faa","Type":"ContainerStarted","Data":"cbfa16c9626c6fa932f51e653df95b5d94be8c28819db41de8c2749eac483ed6"} Nov 03 01:17:46.676735 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:46.676582 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jnvg6" event={"ID":"4c25011b-c85c-4736-9f72-842a96949072","Type":"ContainerStarted","Data":"fddad702658b098eefc83175445d2c4999bc52eea0b4595790ddc5cc9e9559b8"} Nov 03 01:17:46.808664 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:46.808609 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-jnvg6" podStartSLOduration=7.901236111 podStartE2EDuration="21.808590835s" totalImagesPullingTime="13.907354724s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 01:17:25 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:17:46.807668094 +0000 UTC m=+22.894523677" watchObservedRunningTime="2026-11-03 01:17:46.808590835 +0000 UTC m=+22.895446416" Nov 03 01:17:47.311854 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:47.311829 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-jpcqz_bff481c2-d05e-4b45-827d-f5c8a0f60d0b/node-ca/0.log" Nov 03 01:17:47.537197 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:47.537168 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:47.681840 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:47.681759 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n2cjg" event={"ID":"abdd79d9-8d87-4133-85a2-643d26fc9faa","Type":"ContainerStarted","Data":"7de70bb464abf97b482fbdf026d276709d70546c324c38cd2887756032277d3d"} Nov 03 01:17:47.681840 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:47.681827 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n2cjg" event={"ID":"abdd79d9-8d87-4133-85a2-643d26fc9faa","Type":"ContainerStarted","Data":"679a8e9d0c0d1ca2543603e592eef61cf74551f00fe4c86e9bb14fb0fd99e16c"} Nov 03 01:17:47.682355 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:47.681860 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n2cjg" event={"ID":"abdd79d9-8d87-4133-85a2-643d26fc9faa","Type":"ContainerStarted","Data":"a4c712bb77c80486dd200fdfd38548e24ceb5ce82f3fdd9c536fd8b63887a84d"} Nov 03 01:17:47.682355 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:47.681888 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n2cjg" event={"ID":"abdd79d9-8d87-4133-85a2-643d26fc9faa","Type":"ContainerStarted","Data":"f175decfab4f1a9ae5df985ec57b9bee635d3edbb0bf42e2266a7adedd75207b"} Nov 03 01:17:47.682355 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:47.681916 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n2cjg" event={"ID":"abdd79d9-8d87-4133-85a2-643d26fc9faa","Type":"ContainerStarted","Data":"3f947581095565ea00b68f16f5e747bf66523fbf2ce9bedc6ed704b5922729ab"} Nov 03 01:17:47.911897 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:47.911862 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:48.116602 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:48.116575 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:48.420927 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:48.420844 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:48.510530 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:48.510496 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8g56n" Nov 03 01:17:48.510726 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:48.510497 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kg99x" Nov 03 01:17:48.510726 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:17:48.510639 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8g56n" podUID="6ecc2615-b458-41f8-839f-e22c53e7ad0e" Nov 03 01:17:48.510726 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:48.510500 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dfr9t" Nov 03 01:17:48.510726 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:17:48.510678 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kg99x" podUID="3b2bf2f6-28aa-4480-9f4a-f4e7f659e966" Nov 03 01:17:48.510909 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:17:48.510776 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dfr9t" podUID="6179266f-07ff-48cd-bf03-3f1e961529d7" Nov 03 01:17:48.662251 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:48.662217 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:49.274916 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:49.274834 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:49.481920 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:49.481887 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:49.689009 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:49.688971 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n2cjg" event={"ID":"abdd79d9-8d87-4133-85a2-643d26fc9faa","Type":"ContainerStarted","Data":"0ffec3f6f0cffe138b3bae1c613995768e85e9b03e47f7c05aa6d2c330816629"} Nov 03 01:17:49.856305 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:49.856272 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:50.057612 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:50.057581 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:50.510401 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:50.510309 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kg99x" Nov 03 01:17:50.510401 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:50.510380 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8g56n" Nov 03 01:17:50.510401 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:50.510393 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dfr9t" Nov 03 01:17:50.510878 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:17:50.510495 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dfr9t" podUID="6179266f-07ff-48cd-bf03-3f1e961529d7" Nov 03 01:17:50.510878 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:17:50.510549 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kg99x" podUID="3b2bf2f6-28aa-4480-9f4a-f4e7f659e966" Nov 03 01:17:50.510878 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:17:50.510628 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8g56n" podUID="6ecc2615-b458-41f8-839f-e22c53e7ad0e" Nov 03 01:17:50.658480 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:50.658436 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:51.258256 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:51.258073 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:51.458927 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:51.458893 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:51.657788 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:51.657755 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:51.696767 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:51.696739 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n2cjg" event={"ID":"abdd79d9-8d87-4133-85a2-643d26fc9faa","Type":"ContainerStarted","Data":"4e61ece9e64acc38f795633f18361bc3232c70d834aeabe73bf177c133d95430"} Nov 03 01:17:51.696978 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:51.696949 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-n2cjg" Nov 03 01:17:51.697071 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:51.697034 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-n2cjg" Nov 03 01:17:51.712546 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:51.712490 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-n2cjg" Nov 03 01:17:51.889144 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:51.889109 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:51.988895 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:51.988802 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-n2cjg" podStartSLOduration=7.097774135 podStartE2EDuration="26.988789696s" totalImagesPullingTime="19.891015561s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 01:17:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:17:51.937288905 +0000 UTC m=+28.024144487" watchObservedRunningTime="2026-11-03 01:17:51.988789696 +0000 UTC m=+28.075645276" Nov 03 01:17:52.096049 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:52.096016 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:52.510871 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:52.510842 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8g56n" Nov 03 01:17:52.511054 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:52.510903 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dfr9t" Nov 03 01:17:52.511054 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:52.510906 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kg99x" Nov 03 01:17:52.511054 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:17:52.511014 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8g56n" podUID="6ecc2615-b458-41f8-839f-e22c53e7ad0e" Nov 03 01:17:52.511203 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:17:52.511131 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dfr9t" podUID="6179266f-07ff-48cd-bf03-3f1e961529d7" Nov 03 01:17:52.511237 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:17:52.511206 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kg99x" podUID="3b2bf2f6-28aa-4480-9f4a-f4e7f659e966" Nov 03 01:17:52.663015 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:52.662973 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:52.699719 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:52.699694 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-n2cjg" Nov 03 01:17:52.714403 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:52.714360 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-n2cjg" Nov 03 01:17:53.104628 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:53.104600 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:53.301505 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:53.301470 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:53.503921 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:53.503879 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:53.857858 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:53.857828 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:54.058670 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:54.058642 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:54.511652 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:54.511619 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8g56n" Nov 03 01:17:54.511798 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:54.511777 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kg99x" Nov 03 01:17:54.511861 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:17:54.511794 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8g56n" podUID="6ecc2615-b458-41f8-839f-e22c53e7ad0e" Nov 03 01:17:54.511910 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:17:54.511862 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kg99x" podUID="3b2bf2f6-28aa-4480-9f4a-f4e7f659e966" Nov 03 01:17:54.511942 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:54.511908 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dfr9t" Nov 03 01:17:54.511975 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:17:54.511958 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dfr9t" podUID="6179266f-07ff-48cd-bf03-3f1e961529d7" Nov 03 01:17:54.661908 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:54.661873 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:55.104497 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:55.104461 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:55.305422 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:55.305391 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:55.513698 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:55.513619 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:55.855894 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:55.855862 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:56.061938 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:56.061906 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:56.268016 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:56.267947 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:56.511254 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:56.511214 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kg99x" Nov 03 01:17:56.511254 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:56.511245 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dfr9t" Nov 03 01:17:56.511513 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:56.511216 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8g56n" Nov 03 01:17:56.511513 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:17:56.511320 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kg99x" podUID="3b2bf2f6-28aa-4480-9f4a-f4e7f659e966" Nov 03 01:17:56.511513 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:17:56.511458 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8g56n" podUID="6ecc2615-b458-41f8-839f-e22c53e7ad0e" Nov 03 01:17:56.511705 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:17:56.511544 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dfr9t" podUID="6179266f-07ff-48cd-bf03-3f1e961529d7" Nov 03 01:17:56.561800 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:56.561773 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:56.775698 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:56.775664 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:57.269840 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:57.269808 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:57.477609 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:57.477579 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:57.858679 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:57.858642 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:58.061241 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:58.061201 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:58.216575 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:58.216499 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ecc2615-b458-41f8-839f-e22c53e7ad0e-metrics-certs\") pod \"network-metrics-daemon-8g56n\" (UID: \"6ecc2615-b458-41f8-839f-e22c53e7ad0e\") " pod="openshift-multus/network-metrics-daemon-8g56n" Nov 03 01:17:58.216575 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:58.216548 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6179266f-07ff-48cd-bf03-3f1e961529d7-original-pull-secret\") pod \"global-pull-secret-syncer-dfr9t\" (UID: \"6179266f-07ff-48cd-bf03-3f1e961529d7\") " pod="kube-system/global-pull-secret-syncer-dfr9t" Nov 03 01:17:58.216760 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:17:58.216644 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 01:17:58.216760 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:17:58.216646 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 01:17:58.216760 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:17:58.216695 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6179266f-07ff-48cd-bf03-3f1e961529d7-original-pull-secret podName:6179266f-07ff-48cd-bf03-3f1e961529d7 nodeName:}" failed. No retries permitted until 2026-11-03 01:18:30.216681812 +0000 UTC m=+66.303537371 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6179266f-07ff-48cd-bf03-3f1e961529d7-original-pull-secret") pod "global-pull-secret-syncer-dfr9t" (UID: "6179266f-07ff-48cd-bf03-3f1e961529d7") : object "kube-system"/"original-pull-secret" not registered Nov 03 01:17:58.216760 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:17:58.216708 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ecc2615-b458-41f8-839f-e22c53e7ad0e-metrics-certs podName:6ecc2615-b458-41f8-839f-e22c53e7ad0e nodeName:}" failed. No retries permitted until 2026-11-03 01:18:30.216701081 +0000 UTC m=+66.303556640 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6ecc2615-b458-41f8-839f-e22c53e7ad0e-metrics-certs") pod "network-metrics-daemon-8g56n" (UID: "6ecc2615-b458-41f8-839f-e22c53e7ad0e") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 01:17:58.317166 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:58.317136 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgd8w\" (UniqueName: \"kubernetes.io/projected/3b2bf2f6-28aa-4480-9f4a-f4e7f659e966-kube-api-access-vgd8w\") pod \"network-check-target-kg99x\" (UID: \"3b2bf2f6-28aa-4480-9f4a-f4e7f659e966\") " pod="openshift-network-diagnostics/network-check-target-kg99x" Nov 03 01:17:58.317558 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:17:58.317293 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 01:17:58.317558 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:17:58.317312 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 01:17:58.317558 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:17:58.317322 2578 projected.go:196] Error preparing data for projected volume kube-api-access-vgd8w for pod openshift-network-diagnostics/network-check-target-kg99x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 01:17:58.317558 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:17:58.317397 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3b2bf2f6-28aa-4480-9f4a-f4e7f659e966-kube-api-access-vgd8w podName:3b2bf2f6-28aa-4480-9f4a-f4e7f659e966 nodeName:}" failed. No retries permitted until 2026-11-03 01:18:30.31735857 +0000 UTC m=+66.404214133 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-vgd8w" (UniqueName: "kubernetes.io/projected/3b2bf2f6-28aa-4480-9f4a-f4e7f659e966-kube-api-access-vgd8w") pod "network-check-target-kg99x" (UID: "3b2bf2f6-28aa-4480-9f4a-f4e7f659e966") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 01:17:58.459380 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:58.459336 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:58.510721 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:58.510660 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kg99x" Nov 03 01:17:58.510721 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:58.510677 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8g56n" Nov 03 01:17:58.510842 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:58.510762 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dfr9t" Nov 03 01:17:58.510842 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:17:58.510774 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kg99x" podUID="3b2bf2f6-28aa-4480-9f4a-f4e7f659e966" Nov 03 01:17:58.510915 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:17:58.510865 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dfr9t" podUID="6179266f-07ff-48cd-bf03-3f1e961529d7" Nov 03 01:17:58.510915 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:17:58.510900 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8g56n" podUID="6ecc2615-b458-41f8-839f-e22c53e7ad0e" Nov 03 01:17:58.656666 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:17:58.656635 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:00.212083 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:00.212055 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:00.415616 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:00.415585 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:00.511179 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:00.511104 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8g56n" Nov 03 01:18:00.511320 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:00.511111 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dfr9t" Nov 03 01:18:00.511320 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:18:00.511214 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8g56n" podUID="6ecc2615-b458-41f8-839f-e22c53e7ad0e" Nov 03 01:18:00.511320 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:18:00.511304 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dfr9t" podUID="6179266f-07ff-48cd-bf03-3f1e961529d7" Nov 03 01:18:00.511468 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:00.511111 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kg99x" Nov 03 01:18:00.511468 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:18:00.511418 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kg99x" podUID="3b2bf2f6-28aa-4480-9f4a-f4e7f659e966" Nov 03 01:18:00.610440 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:00.610400 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:01.131630 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:01.131593 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:01.476469 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:01.476350 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:01.680132 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:01.680092 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:02.510971 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:02.510939 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dfr9t" Nov 03 01:18:02.511498 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:02.511009 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kg99x" Nov 03 01:18:02.511498 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:02.511031 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8g56n" Nov 03 01:18:02.511498 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:18:02.511128 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kg99x" podUID="3b2bf2f6-28aa-4480-9f4a-f4e7f659e966" Nov 03 01:18:02.511498 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:18:02.511162 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8g56n" podUID="6ecc2615-b458-41f8-839f-e22c53e7ad0e" Nov 03 01:18:02.511498 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:18:02.511246 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dfr9t" podUID="6179266f-07ff-48cd-bf03-3f1e961529d7" Nov 03 01:18:04.217773 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:04.217738 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:04.421688 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:04.421659 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:04.511960 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:04.511858 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kg99x" Nov 03 01:18:04.511960 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:04.511908 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dfr9t" Nov 03 01:18:04.512165 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:04.512003 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8g56n" Nov 03 01:18:04.512165 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:18:04.512004 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kg99x" podUID="3b2bf2f6-28aa-4480-9f4a-f4e7f659e966" Nov 03 01:18:04.512165 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:18:04.512107 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8g56n" podUID="6ecc2615-b458-41f8-839f-e22c53e7ad0e" Nov 03 01:18:04.512281 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:18:04.512203 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dfr9t" podUID="6179266f-07ff-48cd-bf03-3f1e961529d7" Nov 03 01:18:04.746474 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:04.746433 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:04.944480 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:04.944449 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:06.510820 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:06.510780 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8g56n" Nov 03 01:18:06.510820 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:06.510805 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kg99x" Nov 03 01:18:06.511338 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:18:06.510957 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8g56n" podUID="6ecc2615-b458-41f8-839f-e22c53e7ad0e" Nov 03 01:18:06.511338 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:06.510986 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dfr9t" Nov 03 01:18:06.511338 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:18:06.511053 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kg99x" podUID="3b2bf2f6-28aa-4480-9f4a-f4e7f659e966" Nov 03 01:18:06.511338 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:18:06.511131 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dfr9t" podUID="6179266f-07ff-48cd-bf03-3f1e961529d7" Nov 03 01:18:06.752390 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:06.752352 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-dfr9t"] Nov 03 01:18:06.752640 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:06.752623 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dfr9t" Nov 03 01:18:06.752759 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:18:06.752739 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dfr9t" podUID="6179266f-07ff-48cd-bf03-3f1e961529d7" Nov 03 01:18:06.752928 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:06.752904 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-kg99x"] Nov 03 01:18:06.753016 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:06.753002 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kg99x" Nov 03 01:18:06.753124 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:18:06.753103 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kg99x" podUID="3b2bf2f6-28aa-4480-9f4a-f4e7f659e966" Nov 03 01:18:06.753411 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:06.753393 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-8g56n"] Nov 03 01:18:06.753486 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:06.753475 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8g56n" Nov 03 01:18:06.753574 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:18:06.753555 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8g56n" podUID="6ecc2615-b458-41f8-839f-e22c53e7ad0e" Nov 03 01:18:07.480384 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:07.480293 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:07.678555 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:07.678525 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:07.884034 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:07.883856 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:08.087989 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:08.087950 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:08.511123 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:08.511044 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kg99x" Nov 03 01:18:08.511123 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:08.511067 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8g56n" Nov 03 01:18:08.511299 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:08.511047 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dfr9t" Nov 03 01:18:08.511299 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:18:08.511149 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kg99x" podUID="3b2bf2f6-28aa-4480-9f4a-f4e7f659e966" Nov 03 01:18:08.511299 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:18:08.511216 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dfr9t" podUID="6179266f-07ff-48cd-bf03-3f1e961529d7" Nov 03 01:18:08.511299 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:18:08.511287 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8g56n" podUID="6ecc2615-b458-41f8-839f-e22c53e7ad0e" Nov 03 01:18:08.628233 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:08.628204 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:08.829738 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:08.829710 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:09.032975 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:09.032948 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:10.510916 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:10.510887 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dfr9t" Nov 03 01:18:10.510916 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:10.510906 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kg99x" Nov 03 01:18:10.511473 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:10.510939 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8g56n" Nov 03 01:18:10.511473 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:18:10.511011 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kg99x" podUID="3b2bf2f6-28aa-4480-9f4a-f4e7f659e966" Nov 03 01:18:10.511473 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:18:10.511092 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dfr9t" podUID="6179266f-07ff-48cd-bf03-3f1e961529d7" Nov 03 01:18:10.511473 ip-10-0-136-16 kubenswrapper[2578]: E1103 01:18:10.511193 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8g56n" podUID="6ecc2615-b458-41f8-839f-e22c53e7ad0e" Nov 03 01:18:11.257579 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:11.257542 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:11.471680 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:11.471649 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:11.698343 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:11.698318 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-16.us-west-2.compute.internal" event="NodeReady" Nov 03 01:18:11.698826 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:11.698495 2578 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 03 01:18:11.817400 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:11.817352 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-kmpbz"] Nov 03 01:18:11.819245 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:11.819229 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-kmpbz" Nov 03 01:18:11.868905 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:11.868880 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-bbfdw"] Nov 03 01:18:11.870662 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:11.870648 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-4vr4n"] Nov 03 01:18:11.870787 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:11.870766 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-bbfdw" Nov 03 01:18:11.872397 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:11.872356 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-kmpbz"] Nov 03 01:18:11.872487 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:11.872421 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-bbfdw"] Nov 03 01:18:11.872487 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:11.872430 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-4vr4n" Nov 03 01:18:11.873715 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:11.873700 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 03 01:18:11.873766 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:11.873714 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qwk4g\"" Nov 03 01:18:11.873766 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:11.873700 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 03 01:18:11.881464 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:11.881445 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-4vr4n"] Nov 03 01:18:11.884760 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:11.884737 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:11.914984 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:11.914955 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1fdb6f2d-9138-4f57-af12-c3909dff6e5f-config-volume\") pod \"dns-default-kmpbz\" (UID: \"1fdb6f2d-9138-4f57-af12-c3909dff6e5f\") " pod="openshift-dns/dns-default-kmpbz" Nov 03 01:18:11.915091 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:11.914987 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fdb6f2d-9138-4f57-af12-c3909dff6e5f-tmp-dir\") pod \"dns-default-kmpbz\" (UID: \"1fdb6f2d-9138-4f57-af12-c3909dff6e5f\") " pod="openshift-dns/dns-default-kmpbz" Nov 03 01:18:11.915091 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:11.915022 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8flcc\" (UniqueName: \"kubernetes.io/projected/1fdb6f2d-9138-4f57-af12-c3909dff6e5f-kube-api-access-8flcc\") pod \"dns-default-kmpbz\" (UID: \"1fdb6f2d-9138-4f57-af12-c3909dff6e5f\") " pod="openshift-dns/dns-default-kmpbz" Nov 03 01:18:11.915205 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:11.915190 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1fdb6f2d-9138-4f57-af12-c3909dff6e5f-metrics-tls\") pod \"dns-default-kmpbz\" (UID: \"1fdb6f2d-9138-4f57-af12-c3909dff6e5f\") " pod="openshift-dns/dns-default-kmpbz" Nov 03 01:18:11.974085 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:11.973967 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 03 01:18:11.974085 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:11.973987 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 03 01:18:11.974085 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:11.973983 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 03 01:18:11.974085 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:11.974013 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 03 01:18:11.974085 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:11.974039 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 03 01:18:11.974085 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:11.974045 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 03 01:18:11.974417 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:11.974123 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-9f8wp\"" Nov 03 01:18:11.974417 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:11.974139 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 03 01:18:11.974417 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:11.974201 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-r69hg\"" Nov 03 01:18:12.016441 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:12.016415 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2320cce9-0fdb-470a-ae01-e88118ba0419-crio-socket\") pod \"insights-runtime-extractor-bbfdw\" (UID: \"2320cce9-0fdb-470a-ae01-e88118ba0419\") " pod="openshift-insights/insights-runtime-extractor-bbfdw" Nov 03 01:18:12.016569 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:12.016451 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8flcc\" (UniqueName: \"kubernetes.io/projected/1fdb6f2d-9138-4f57-af12-c3909dff6e5f-kube-api-access-8flcc\") pod \"dns-default-kmpbz\" (UID: \"1fdb6f2d-9138-4f57-af12-c3909dff6e5f\") " pod="openshift-dns/dns-default-kmpbz" Nov 03 01:18:12.016569 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:12.016489 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2320cce9-0fdb-470a-ae01-e88118ba0419-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bbfdw\" (UID: \"2320cce9-0fdb-470a-ae01-e88118ba0419\") " pod="openshift-insights/insights-runtime-extractor-bbfdw" Nov 03 01:18:12.016673 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:12.016620 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/97cb83aa-8922-4221-8407-fbbb6c39f5c7-cert\") pod \"ingress-canary-4vr4n\" (UID: \"97cb83aa-8922-4221-8407-fbbb6c39f5c7\") " pod="openshift-ingress-canary/ingress-canary-4vr4n" Nov 03 01:18:12.016724 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:12.016683 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1fdb6f2d-9138-4f57-af12-c3909dff6e5f-metrics-tls\") pod \"dns-default-kmpbz\" (UID: \"1fdb6f2d-9138-4f57-af12-c3909dff6e5f\") " pod="openshift-dns/dns-default-kmpbz" Nov 03 01:18:12.016724 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:12.016713 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1fdb6f2d-9138-4f57-af12-c3909dff6e5f-config-volume\") pod \"dns-default-kmpbz\" (UID: \"1fdb6f2d-9138-4f57-af12-c3909dff6e5f\") " pod="openshift-dns/dns-default-kmpbz" Nov 03 01:18:12.016814 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:12.016730 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2cz9\" (UniqueName: \"kubernetes.io/projected/2320cce9-0fdb-470a-ae01-e88118ba0419-kube-api-access-b2cz9\") pod \"insights-runtime-extractor-bbfdw\" (UID: \"2320cce9-0fdb-470a-ae01-e88118ba0419\") " pod="openshift-insights/insights-runtime-extractor-bbfdw" Nov 03 01:18:12.016814 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:12.016747 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fdb6f2d-9138-4f57-af12-c3909dff6e5f-tmp-dir\") pod \"dns-default-kmpbz\" (UID: \"1fdb6f2d-9138-4f57-af12-c3909dff6e5f\") " pod="openshift-dns/dns-default-kmpbz" Nov 03 01:18:12.016814 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:12.016761 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2320cce9-0fdb-470a-ae01-e88118ba0419-data-volume\") pod \"insights-runtime-extractor-bbfdw\" (UID: \"2320cce9-0fdb-470a-ae01-e88118ba0419\") " pod="openshift-insights/insights-runtime-extractor-bbfdw" Nov 03 01:18:12.016945 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:12.016814 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4v9cn\" (UniqueName: \"kubernetes.io/projected/97cb83aa-8922-4221-8407-fbbb6c39f5c7-kube-api-access-4v9cn\") pod \"ingress-canary-4vr4n\" (UID: \"97cb83aa-8922-4221-8407-fbbb6c39f5c7\") " pod="openshift-ingress-canary/ingress-canary-4vr4n" Nov 03 01:18:12.016945 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:12.016849 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2320cce9-0fdb-470a-ae01-e88118ba0419-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bbfdw\" (UID: \"2320cce9-0fdb-470a-ae01-e88118ba0419\") " pod="openshift-insights/insights-runtime-extractor-bbfdw" Nov 03 01:18:12.017096 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:12.017076 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fdb6f2d-9138-4f57-af12-c3909dff6e5f-tmp-dir\") pod \"dns-default-kmpbz\" (UID: \"1fdb6f2d-9138-4f57-af12-c3909dff6e5f\") " pod="openshift-dns/dns-default-kmpbz" Nov 03 01:18:12.017287 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:12.017268 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1fdb6f2d-9138-4f57-af12-c3909dff6e5f-config-volume\") pod \"dns-default-kmpbz\" (UID: \"1fdb6f2d-9138-4f57-af12-c3909dff6e5f\") " pod="openshift-dns/dns-default-kmpbz" Nov 03 01:18:12.020614 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:12.020593 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1fdb6f2d-9138-4f57-af12-c3909dff6e5f-metrics-tls\") pod \"dns-default-kmpbz\" (UID: \"1fdb6f2d-9138-4f57-af12-c3909dff6e5f\") " pod="openshift-dns/dns-default-kmpbz" Nov 03 01:18:12.078410 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:12.078384 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8flcc\" (UniqueName: \"kubernetes.io/projected/1fdb6f2d-9138-4f57-af12-c3909dff6e5f-kube-api-access-8flcc\") pod \"dns-default-kmpbz\" (UID: \"1fdb6f2d-9138-4f57-af12-c3909dff6e5f\") " pod="openshift-dns/dns-default-kmpbz" Nov 03 01:18:12.083656 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:12.083622 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:12.120289 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:12.117630 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2320cce9-0fdb-470a-ae01-e88118ba0419-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bbfdw\" (UID: \"2320cce9-0fdb-470a-ae01-e88118ba0419\") " pod="openshift-insights/insights-runtime-extractor-bbfdw" Nov 03 01:18:12.120289 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:12.117735 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/97cb83aa-8922-4221-8407-fbbb6c39f5c7-cert\") pod \"ingress-canary-4vr4n\" (UID: \"97cb83aa-8922-4221-8407-fbbb6c39f5c7\") " pod="openshift-ingress-canary/ingress-canary-4vr4n" Nov 03 01:18:12.120289 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:12.117818 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2cz9\" (UniqueName: \"kubernetes.io/projected/2320cce9-0fdb-470a-ae01-e88118ba0419-kube-api-access-b2cz9\") pod \"insights-runtime-extractor-bbfdw\" (UID: \"2320cce9-0fdb-470a-ae01-e88118ba0419\") " pod="openshift-insights/insights-runtime-extractor-bbfdw" Nov 03 01:18:12.120289 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:12.117869 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2320cce9-0fdb-470a-ae01-e88118ba0419-data-volume\") pod \"insights-runtime-extractor-bbfdw\" (UID: \"2320cce9-0fdb-470a-ae01-e88118ba0419\") " pod="openshift-insights/insights-runtime-extractor-bbfdw" Nov 03 01:18:12.120289 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:12.117905 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4v9cn\" (UniqueName: \"kubernetes.io/projected/97cb83aa-8922-4221-8407-fbbb6c39f5c7-kube-api-access-4v9cn\") pod \"ingress-canary-4vr4n\" (UID: \"97cb83aa-8922-4221-8407-fbbb6c39f5c7\") " pod="openshift-ingress-canary/ingress-canary-4vr4n" Nov 03 01:18:12.120289 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:12.117961 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2320cce9-0fdb-470a-ae01-e88118ba0419-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bbfdw\" (UID: \"2320cce9-0fdb-470a-ae01-e88118ba0419\") " pod="openshift-insights/insights-runtime-extractor-bbfdw" Nov 03 01:18:12.120289 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:12.118024 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2320cce9-0fdb-470a-ae01-e88118ba0419-crio-socket\") pod \"insights-runtime-extractor-bbfdw\" (UID: \"2320cce9-0fdb-470a-ae01-e88118ba0419\") " pod="openshift-insights/insights-runtime-extractor-bbfdw" Nov 03 01:18:12.120289 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:12.118288 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2320cce9-0fdb-470a-ae01-e88118ba0419-crio-socket\") pod \"insights-runtime-extractor-bbfdw\" (UID: \"2320cce9-0fdb-470a-ae01-e88118ba0419\") " pod="openshift-insights/insights-runtime-extractor-bbfdw" Nov 03 01:18:12.120289 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:12.119757 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2320cce9-0fdb-470a-ae01-e88118ba0419-data-volume\") pod \"insights-runtime-extractor-bbfdw\" (UID: \"2320cce9-0fdb-470a-ae01-e88118ba0419\") " pod="openshift-insights/insights-runtime-extractor-bbfdw" Nov 03 01:18:12.120830 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:12.120431 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2320cce9-0fdb-470a-ae01-e88118ba0419-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bbfdw\" (UID: \"2320cce9-0fdb-470a-ae01-e88118ba0419\") " pod="openshift-insights/insights-runtime-extractor-bbfdw" Nov 03 01:18:12.121766 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:12.121745 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2320cce9-0fdb-470a-ae01-e88118ba0419-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bbfdw\" (UID: \"2320cce9-0fdb-470a-ae01-e88118ba0419\") " pod="openshift-insights/insights-runtime-extractor-bbfdw" Nov 03 01:18:12.122299 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:12.122278 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/97cb83aa-8922-4221-8407-fbbb6c39f5c7-cert\") pod \"ingress-canary-4vr4n\" (UID: \"97cb83aa-8922-4221-8407-fbbb6c39f5c7\") " pod="openshift-ingress-canary/ingress-canary-4vr4n" Nov 03 01:18:12.130305 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:12.130287 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-kmpbz" Nov 03 01:18:12.182516 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:12.182454 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2cz9\" (UniqueName: \"kubernetes.io/projected/2320cce9-0fdb-470a-ae01-e88118ba0419-kube-api-access-b2cz9\") pod \"insights-runtime-extractor-bbfdw\" (UID: \"2320cce9-0fdb-470a-ae01-e88118ba0419\") " pod="openshift-insights/insights-runtime-extractor-bbfdw" Nov 03 01:18:12.182680 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:12.182657 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4v9cn\" (UniqueName: \"kubernetes.io/projected/97cb83aa-8922-4221-8407-fbbb6c39f5c7-kube-api-access-4v9cn\") pod \"ingress-canary-4vr4n\" (UID: \"97cb83aa-8922-4221-8407-fbbb6c39f5c7\") " pod="openshift-ingress-canary/ingress-canary-4vr4n" Nov 03 01:18:12.184301 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:12.184281 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-bbfdw" Nov 03 01:18:12.189215 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:12.187072 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-4vr4n" Nov 03 01:18:12.502897 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:18:12.502869 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fdb6f2d_9138_4f57_af12_c3909dff6e5f.slice/crio-8faf3e8cf655ef04445ca37b09e02a7a6303ffbe2ea86a0fd98a8e918838d88d WatchSource:0}: Error finding container 8faf3e8cf655ef04445ca37b09e02a7a6303ffbe2ea86a0fd98a8e918838d88d: Status 404 returned error can't find the container with id 8faf3e8cf655ef04445ca37b09e02a7a6303ffbe2ea86a0fd98a8e918838d88d Nov 03 01:18:12.504744 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:12.504722 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-kmpbz"] Nov 03 01:18:12.510767 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:12.510746 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8g56n" Nov 03 01:18:12.510767 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:12.510756 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dfr9t" Nov 03 01:18:12.510914 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:12.510884 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kg99x" Nov 03 01:18:12.564885 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:12.564860 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 01:18:12.565024 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:12.564963 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 01:18:12.565150 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:12.565133 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-mmkcs\"" Nov 03 01:18:12.565295 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:12.565279 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-fqzf6\"" Nov 03 01:18:12.565295 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:12.565291 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 01:18:12.565356 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:12.565311 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 01:18:12.565356 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:12.565343 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-7sd2v\"" Nov 03 01:18:12.606885 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:18:12.606860 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97cb83aa_8922_4221_8407_fbbb6c39f5c7.slice/crio-7c7251f3fe76aa5de4ec3a3488313d38429bb12e51be59710fb3d62113862144 WatchSource:0}: Error finding container 7c7251f3fe76aa5de4ec3a3488313d38429bb12e51be59710fb3d62113862144: Status 404 returned error can't find the container with id 7c7251f3fe76aa5de4ec3a3488313d38429bb12e51be59710fb3d62113862144 Nov 03 01:18:12.609287 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:12.609265 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-4vr4n"] Nov 03 01:18:12.611771 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:18:12.611729 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2320cce9_0fdb_470a_ae01_e88118ba0419.slice/crio-beaf53b14654cf966260cc8f6b640829a5f2cbd5414b1aa7902e5fb960ffd8fb WatchSource:0}: Error finding container beaf53b14654cf966260cc8f6b640829a5f2cbd5414b1aa7902e5fb960ffd8fb: Status 404 returned error can't find the container with id beaf53b14654cf966260cc8f6b640829a5f2cbd5414b1aa7902e5fb960ffd8fb Nov 03 01:18:12.617677 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:12.617656 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-bbfdw"] Nov 03 01:18:12.742927 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:12.742895 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-4vr4n" event={"ID":"97cb83aa-8922-4221-8407-fbbb6c39f5c7","Type":"ContainerStarted","Data":"7c7251f3fe76aa5de4ec3a3488313d38429bb12e51be59710fb3d62113862144"} Nov 03 01:18:12.744404 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:12.744380 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bbfdw" event={"ID":"2320cce9-0fdb-470a-ae01-e88118ba0419","Type":"ContainerStarted","Data":"9fb057ccfdb9b2325d6d08f27cdbddfba3ed7b5bcfbbbafb2536aa817793be20"} Nov 03 01:18:12.744523 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:12.744432 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bbfdw" event={"ID":"2320cce9-0fdb-470a-ae01-e88118ba0419","Type":"ContainerStarted","Data":"beaf53b14654cf966260cc8f6b640829a5f2cbd5414b1aa7902e5fb960ffd8fb"} Nov 03 01:18:12.745321 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:12.745302 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kmpbz" event={"ID":"1fdb6f2d-9138-4f57-af12-c3909dff6e5f","Type":"ContainerStarted","Data":"8faf3e8cf655ef04445ca37b09e02a7a6303ffbe2ea86a0fd98a8e918838d88d"} Nov 03 01:18:12.954326 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:12.954295 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:13.158473 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:13.158445 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:13.696533 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:13.696480 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:14.753875 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:14.753835 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bbfdw" event={"ID":"2320cce9-0fdb-470a-ae01-e88118ba0419","Type":"ContainerStarted","Data":"98a09b8751bca085ffe940ef1e02b681d510d745cbf5a1a083d1645b15d11b18"} Nov 03 01:18:14.807890 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:14.807850 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:15.023028 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:15.022913 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:15.303993 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:15.303958 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:15.403714 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:15.403682 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-lcwkf"] Nov 03 01:18:15.405994 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:15.405973 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lcwkf" Nov 03 01:18:15.463279 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:15.462392 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 03 01:18:15.515396 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:15.513224 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 03 01:18:15.515396 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:15.513337 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 03 01:18:15.515396 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:15.513442 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-4pcs8\"" Nov 03 01:18:15.515396 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:15.513450 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Nov 03 01:18:15.515396 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:15.513508 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Nov 03 01:18:15.515396 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:15.514328 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-lcwkf"] Nov 03 01:18:15.515396 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:15.514896 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-htfx6"] Nov 03 01:18:15.517019 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:15.516999 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-htfx6" Nov 03 01:18:15.533337 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:15.533286 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-5z5ct"] Nov 03 01:18:15.535345 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:15.535312 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:15.535571 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:15.535556 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-5z5ct" Nov 03 01:18:15.542297 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:15.542276 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0e35c4cd-ef18-40c1-8568-f77773e378b2-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-lcwkf\" (UID: \"0e35c4cd-ef18-40c1-8568-f77773e378b2\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lcwkf" Nov 03 01:18:15.542428 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:15.542310 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0e35c4cd-ef18-40c1-8568-f77773e378b2-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-lcwkf\" (UID: \"0e35c4cd-ef18-40c1-8568-f77773e378b2\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lcwkf" Nov 03 01:18:15.542584 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:15.542467 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shwrh\" (UniqueName: \"kubernetes.io/projected/0e35c4cd-ef18-40c1-8568-f77773e378b2-kube-api-access-shwrh\") pod \"openshift-state-metrics-7bd7489957-lcwkf\" (UID: \"0e35c4cd-ef18-40c1-8568-f77773e378b2\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lcwkf" Nov 03 01:18:15.542584 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:15.542515 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0e35c4cd-ef18-40c1-8568-f77773e378b2-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-lcwkf\" (UID: \"0e35c4cd-ef18-40c1-8568-f77773e378b2\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lcwkf" Nov 03 01:18:15.571776 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:15.571735 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-htfx6"] Nov 03 01:18:15.574287 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:15.574263 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Nov 03 01:18:15.578050 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:15.577974 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Nov 03 01:18:15.578050 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:15.577985 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-w97sx\"" Nov 03 01:18:15.578050 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:15.578017 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Nov 03 01:18:15.643266 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:15.643226 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/30eede54-1970-4f52-b219-70d3c469ddd4-node-exporter-tls\") pod \"node-exporter-5z5ct\" (UID: \"30eede54-1970-4f52-b219-70d3c469ddd4\") " pod="openshift-monitoring/node-exporter-5z5ct" Nov 03 01:18:15.643476 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:15.643277 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shwrh\" (UniqueName: \"kubernetes.io/projected/0e35c4cd-ef18-40c1-8568-f77773e378b2-kube-api-access-shwrh\") pod \"openshift-state-metrics-7bd7489957-lcwkf\" (UID: \"0e35c4cd-ef18-40c1-8568-f77773e378b2\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lcwkf" Nov 03 01:18:15.643476 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:15.643353 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0e35c4cd-ef18-40c1-8568-f77773e378b2-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-lcwkf\" (UID: \"0e35c4cd-ef18-40c1-8568-f77773e378b2\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lcwkf" Nov 03 01:18:15.643597 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:15.643486 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/766a044f-fc66-4998-8006-343c0329860d-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-htfx6\" (UID: \"766a044f-fc66-4998-8006-343c0329860d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-htfx6" Nov 03 01:18:15.643648 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:15.643600 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/766a044f-fc66-4998-8006-343c0329860d-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-htfx6\" (UID: \"766a044f-fc66-4998-8006-343c0329860d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-htfx6" Nov 03 01:18:15.643701 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:15.643664 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/766a044f-fc66-4998-8006-343c0329860d-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-htfx6\" (UID: \"766a044f-fc66-4998-8006-343c0329860d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-htfx6" Nov 03 01:18:15.643701 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:15.643691 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/30eede54-1970-4f52-b219-70d3c469ddd4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5z5ct\" (UID: \"30eede54-1970-4f52-b219-70d3c469ddd4\") " pod="openshift-monitoring/node-exporter-5z5ct" Nov 03 01:18:15.643794 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:15.643711 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0e35c4cd-ef18-40c1-8568-f77773e378b2-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-lcwkf\" (UID: \"0e35c4cd-ef18-40c1-8568-f77773e378b2\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lcwkf" Nov 03 01:18:15.643794 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:15.643727 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0e35c4cd-ef18-40c1-8568-f77773e378b2-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-lcwkf\" (UID: \"0e35c4cd-ef18-40c1-8568-f77773e378b2\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lcwkf" Nov 03 01:18:15.643794 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:15.643753 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7ckb\" (UniqueName: \"kubernetes.io/projected/766a044f-fc66-4998-8006-343c0329860d-kube-api-access-b7ckb\") pod \"kube-state-metrics-5dc8d7d89b-htfx6\" (UID: \"766a044f-fc66-4998-8006-343c0329860d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-htfx6" Nov 03 01:18:15.643794 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:15.643777 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/30eede54-1970-4f52-b219-70d3c469ddd4-sys\") pod \"node-exporter-5z5ct\" (UID: \"30eede54-1970-4f52-b219-70d3c469ddd4\") " pod="openshift-monitoring/node-exporter-5z5ct" Nov 03 01:18:15.643981 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:15.643802 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/30eede54-1970-4f52-b219-70d3c469ddd4-node-exporter-textfile\") pod \"node-exporter-5z5ct\" (UID: \"30eede54-1970-4f52-b219-70d3c469ddd4\") " pod="openshift-monitoring/node-exporter-5z5ct" Nov 03 01:18:15.643981 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:15.643828 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sq5fx\" (UniqueName: \"kubernetes.io/projected/30eede54-1970-4f52-b219-70d3c469ddd4-kube-api-access-sq5fx\") pod \"node-exporter-5z5ct\" (UID: \"30eede54-1970-4f52-b219-70d3c469ddd4\") " pod="openshift-monitoring/node-exporter-5z5ct" Nov 03 01:18:15.643981 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:15.643855 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/30eede54-1970-4f52-b219-70d3c469ddd4-node-exporter-wtmp\") pod \"node-exporter-5z5ct\" (UID: \"30eede54-1970-4f52-b219-70d3c469ddd4\") " pod="openshift-monitoring/node-exporter-5z5ct" Nov 03 01:18:15.643981 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:15.643879 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/30eede54-1970-4f52-b219-70d3c469ddd4-node-exporter-accelerators-collector-config\") pod \"node-exporter-5z5ct\" (UID: \"30eede54-1970-4f52-b219-70d3c469ddd4\") " pod="openshift-monitoring/node-exporter-5z5ct" Nov 03 01:18:15.643981 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:15.643902 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/766a044f-fc66-4998-8006-343c0329860d-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-htfx6\" (UID: \"766a044f-fc66-4998-8006-343c0329860d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-htfx6" Nov 03 01:18:15.643981 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:15.643937 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/30eede54-1970-4f52-b219-70d3c469ddd4-root\") pod \"node-exporter-5z5ct\" (UID: \"30eede54-1970-4f52-b219-70d3c469ddd4\") " pod="openshift-monitoring/node-exporter-5z5ct" Nov 03 01:18:15.643981 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:15.643956 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/766a044f-fc66-4998-8006-343c0329860d-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-htfx6\" (UID: \"766a044f-fc66-4998-8006-343c0329860d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-htfx6" Nov 03 01:18:15.644305 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:15.643987 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/30eede54-1970-4f52-b219-70d3c469ddd4-metrics-client-ca\") pod \"node-exporter-5z5ct\" (UID: \"30eede54-1970-4f52-b219-70d3c469ddd4\") " pod="openshift-monitoring/node-exporter-5z5ct" Nov 03 01:18:15.646032 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:15.646008 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0e35c4cd-ef18-40c1-8568-f77773e378b2-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-lcwkf\" (UID: \"0e35c4cd-ef18-40c1-8568-f77773e378b2\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lcwkf" Nov 03 01:18:15.647222 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:15.647200 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0e35c4cd-ef18-40c1-8568-f77773e378b2-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-lcwkf\" (UID: \"0e35c4cd-ef18-40c1-8568-f77773e378b2\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lcwkf" Nov 03 01:18:15.649407 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:15.649386 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0e35c4cd-ef18-40c1-8568-f77773e378b2-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-lcwkf\" (UID: \"0e35c4cd-ef18-40c1-8568-f77773e378b2\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lcwkf" Nov 03 01:18:15.674448 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:15.674422 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 03 01:18:15.677725 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:15.677699 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 03 01:18:15.677936 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:15.677913 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 03 01:18:15.678043 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:15.678027 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-w7g4c\"" Nov 03 01:18:15.706461 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:15.706434 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shwrh\" (UniqueName: \"kubernetes.io/projected/0e35c4cd-ef18-40c1-8568-f77773e378b2-kube-api-access-shwrh\") pod \"openshift-state-metrics-7bd7489957-lcwkf\" (UID: \"0e35c4cd-ef18-40c1-8568-f77773e378b2\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lcwkf" Nov 03 01:18:15.720211 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:15.720177 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lcwkf" Nov 03 01:18:15.743307 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:15.743269 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:15.744326 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:15.744302 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7ckb\" (UniqueName: \"kubernetes.io/projected/766a044f-fc66-4998-8006-343c0329860d-kube-api-access-b7ckb\") pod \"kube-state-metrics-5dc8d7d89b-htfx6\" (UID: \"766a044f-fc66-4998-8006-343c0329860d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-htfx6" Nov 03 01:18:15.744464 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:15.744345 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/30eede54-1970-4f52-b219-70d3c469ddd4-sys\") pod \"node-exporter-5z5ct\" (UID: \"30eede54-1970-4f52-b219-70d3c469ddd4\") " pod="openshift-monitoring/node-exporter-5z5ct" Nov 03 01:18:15.744464 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:15.744390 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/30eede54-1970-4f52-b219-70d3c469ddd4-node-exporter-textfile\") pod \"node-exporter-5z5ct\" (UID: \"30eede54-1970-4f52-b219-70d3c469ddd4\") " pod="openshift-monitoring/node-exporter-5z5ct" Nov 03 01:18:15.744464 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:15.744415 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sq5fx\" (UniqueName: \"kubernetes.io/projected/30eede54-1970-4f52-b219-70d3c469ddd4-kube-api-access-sq5fx\") pod \"node-exporter-5z5ct\" (UID: \"30eede54-1970-4f52-b219-70d3c469ddd4\") " pod="openshift-monitoring/node-exporter-5z5ct" Nov 03 01:18:15.744464 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:15.744433 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/30eede54-1970-4f52-b219-70d3c469ddd4-node-exporter-wtmp\") pod \"node-exporter-5z5ct\" (UID: \"30eede54-1970-4f52-b219-70d3c469ddd4\") " pod="openshift-monitoring/node-exporter-5z5ct" Nov 03 01:18:15.744464 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:15.744437 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/30eede54-1970-4f52-b219-70d3c469ddd4-sys\") pod \"node-exporter-5z5ct\" (UID: \"30eede54-1970-4f52-b219-70d3c469ddd4\") " pod="openshift-monitoring/node-exporter-5z5ct" Nov 03 01:18:15.744464 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:15.744449 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/30eede54-1970-4f52-b219-70d3c469ddd4-node-exporter-accelerators-collector-config\") pod \"node-exporter-5z5ct\" (UID: \"30eede54-1970-4f52-b219-70d3c469ddd4\") " pod="openshift-monitoring/node-exporter-5z5ct" Nov 03 01:18:15.744754 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:15.744470 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/766a044f-fc66-4998-8006-343c0329860d-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-htfx6\" (UID: \"766a044f-fc66-4998-8006-343c0329860d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-htfx6" Nov 03 01:18:15.744754 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:15.744501 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/30eede54-1970-4f52-b219-70d3c469ddd4-root\") pod \"node-exporter-5z5ct\" (UID: \"30eede54-1970-4f52-b219-70d3c469ddd4\") " pod="openshift-monitoring/node-exporter-5z5ct" Nov 03 01:18:15.744754 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:15.744521 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/766a044f-fc66-4998-8006-343c0329860d-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-htfx6\" (UID: \"766a044f-fc66-4998-8006-343c0329860d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-htfx6" Nov 03 01:18:15.744754 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:15.744539 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/30eede54-1970-4f52-b219-70d3c469ddd4-metrics-client-ca\") pod \"node-exporter-5z5ct\" (UID: \"30eede54-1970-4f52-b219-70d3c469ddd4\") " pod="openshift-monitoring/node-exporter-5z5ct" Nov 03 01:18:15.744754 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:15.744564 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/30eede54-1970-4f52-b219-70d3c469ddd4-node-exporter-tls\") pod \"node-exporter-5z5ct\" (UID: \"30eede54-1970-4f52-b219-70d3c469ddd4\") " pod="openshift-monitoring/node-exporter-5z5ct" Nov 03 01:18:15.744754 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:15.744601 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/766a044f-fc66-4998-8006-343c0329860d-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-htfx6\" (UID: \"766a044f-fc66-4998-8006-343c0329860d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-htfx6" Nov 03 01:18:15.744754 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:15.744639 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/766a044f-fc66-4998-8006-343c0329860d-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-htfx6\" (UID: \"766a044f-fc66-4998-8006-343c0329860d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-htfx6" Nov 03 01:18:15.744754 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:15.744675 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/766a044f-fc66-4998-8006-343c0329860d-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-htfx6\" (UID: \"766a044f-fc66-4998-8006-343c0329860d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-htfx6" Nov 03 01:18:15.744754 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:15.744708 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/30eede54-1970-4f52-b219-70d3c469ddd4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5z5ct\" (UID: \"30eede54-1970-4f52-b219-70d3c469ddd4\") " pod="openshift-monitoring/node-exporter-5z5ct" Nov 03 01:18:15.745176 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:15.744863 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/30eede54-1970-4f52-b219-70d3c469ddd4-node-exporter-textfile\") pod \"node-exporter-5z5ct\" (UID: \"30eede54-1970-4f52-b219-70d3c469ddd4\") " pod="openshift-monitoring/node-exporter-5z5ct" Nov 03 01:18:15.745612 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:15.745319 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/766a044f-fc66-4998-8006-343c0329860d-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-htfx6\" (UID: \"766a044f-fc66-4998-8006-343c0329860d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-htfx6" Nov 03 01:18:15.745612 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:15.745384 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/766a044f-fc66-4998-8006-343c0329860d-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-htfx6\" (UID: \"766a044f-fc66-4998-8006-343c0329860d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-htfx6" Nov 03 01:18:15.745612 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:15.745452 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/30eede54-1970-4f52-b219-70d3c469ddd4-root\") pod \"node-exporter-5z5ct\" (UID: \"30eede54-1970-4f52-b219-70d3c469ddd4\") " pod="openshift-monitoring/node-exporter-5z5ct" Nov 03 01:18:15.745612 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:15.745483 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/30eede54-1970-4f52-b219-70d3c469ddd4-metrics-client-ca\") pod \"node-exporter-5z5ct\" (UID: \"30eede54-1970-4f52-b219-70d3c469ddd4\") " pod="openshift-monitoring/node-exporter-5z5ct" Nov 03 01:18:15.745612 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:15.745574 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/30eede54-1970-4f52-b219-70d3c469ddd4-node-exporter-wtmp\") pod \"node-exporter-5z5ct\" (UID: \"30eede54-1970-4f52-b219-70d3c469ddd4\") " pod="openshift-monitoring/node-exporter-5z5ct" Nov 03 01:18:15.745981 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:15.745948 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/30eede54-1970-4f52-b219-70d3c469ddd4-node-exporter-accelerators-collector-config\") pod \"node-exporter-5z5ct\" (UID: \"30eede54-1970-4f52-b219-70d3c469ddd4\") " pod="openshift-monitoring/node-exporter-5z5ct" Nov 03 01:18:15.746186 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:15.746162 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/766a044f-fc66-4998-8006-343c0329860d-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-htfx6\" (UID: \"766a044f-fc66-4998-8006-343c0329860d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-htfx6" Nov 03 01:18:15.747997 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:15.747977 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/766a044f-fc66-4998-8006-343c0329860d-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-htfx6\" (UID: \"766a044f-fc66-4998-8006-343c0329860d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-htfx6" Nov 03 01:18:15.748098 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:15.748053 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/766a044f-fc66-4998-8006-343c0329860d-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-htfx6\" (UID: \"766a044f-fc66-4998-8006-343c0329860d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-htfx6" Nov 03 01:18:15.749276 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:15.749258 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/30eede54-1970-4f52-b219-70d3c469ddd4-node-exporter-tls\") pod \"node-exporter-5z5ct\" (UID: \"30eede54-1970-4f52-b219-70d3c469ddd4\") " pod="openshift-monitoring/node-exporter-5z5ct" Nov 03 01:18:15.749391 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:15.749340 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/30eede54-1970-4f52-b219-70d3c469ddd4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5z5ct\" (UID: \"30eede54-1970-4f52-b219-70d3c469ddd4\") " pod="openshift-monitoring/node-exporter-5z5ct" Nov 03 01:18:15.759749 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:15.759711 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kmpbz" event={"ID":"1fdb6f2d-9138-4f57-af12-c3909dff6e5f","Type":"ContainerStarted","Data":"4f8c200929364cf003c9378f0a4f45d0c119ef20515b33fe21380817544ffec4"} Nov 03 01:18:15.760114 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:15.759786 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kmpbz" event={"ID":"1fdb6f2d-9138-4f57-af12-c3909dff6e5f","Type":"ContainerStarted","Data":"4c86a6c260b859b479387e4add66a6be67d68069e1bb38db80d95b03f9bc6a3a"} Nov 03 01:18:15.760458 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:15.760434 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-kmpbz" Nov 03 01:18:15.808984 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:15.808850 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7ckb\" (UniqueName: \"kubernetes.io/projected/766a044f-fc66-4998-8006-343c0329860d-kube-api-access-b7ckb\") pod \"kube-state-metrics-5dc8d7d89b-htfx6\" (UID: \"766a044f-fc66-4998-8006-343c0329860d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-htfx6" Nov 03 01:18:15.808984 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:15.808941 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sq5fx\" (UniqueName: \"kubernetes.io/projected/30eede54-1970-4f52-b219-70d3c469ddd4-kube-api-access-sq5fx\") pod \"node-exporter-5z5ct\" (UID: \"30eede54-1970-4f52-b219-70d3c469ddd4\") " pod="openshift-monitoring/node-exporter-5z5ct" Nov 03 01:18:15.830776 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:15.830742 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-htfx6" Nov 03 01:18:15.849882 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:15.849851 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-5z5ct" Nov 03 01:18:15.866500 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:18:15.866467 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30eede54_1970_4f52_b219_70d3c469ddd4.slice/crio-0e9e7edee56b546df74773696b5a5d39e39e5a667ed6d072684745e5f19334ff WatchSource:0}: Error finding container 0e9e7edee56b546df74773696b5a5d39e39e5a667ed6d072684745e5f19334ff: Status 404 returned error can't find the container with id 0e9e7edee56b546df74773696b5a5d39e39e5a667ed6d072684745e5f19334ff Nov 03 01:18:15.936305 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:15.926671 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-kmpbz" podStartSLOduration=2.252720137 podStartE2EDuration="4.926654562s" totalImagesPullingTime="2.673934425s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:18:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:18:15.924342311 +0000 UTC m=+52.011197894" watchObservedRunningTime="2026-11-03 01:18:15.926654562 +0000 UTC m=+52.013510148" Nov 03 01:18:15.950881 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:15.950832 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:16.146390 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:16.146346 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-lcwkf"] Nov 03 01:18:16.147093 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:18:16.147069 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e35c4cd_ef18_40c1_8568_f77773e378b2.slice/crio-a8e0819cc9cee4743891c1bd3a02cdedd551a431d84689742656f35cef6c34de WatchSource:0}: Error finding container a8e0819cc9cee4743891c1bd3a02cdedd551a431d84689742656f35cef6c34de: Status 404 returned error can't find the container with id a8e0819cc9cee4743891c1bd3a02cdedd551a431d84689742656f35cef6c34de Nov 03 01:18:16.153355 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:16.153332 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:16.238474 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:16.238446 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-htfx6"] Nov 03 01:18:16.764762 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:16.764733 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-4vr4n" event={"ID":"97cb83aa-8922-4221-8407-fbbb6c39f5c7","Type":"ContainerStarted","Data":"a2ee292cebf000a451b1fa84417b1f29b427449cdb6bf66f0fa7970fb2e0ad96"} Nov 03 01:18:16.766005 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:16.765982 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5z5ct" event={"ID":"30eede54-1970-4f52-b219-70d3c469ddd4","Type":"ContainerStarted","Data":"0e9e7edee56b546df74773696b5a5d39e39e5a667ed6d072684745e5f19334ff"} Nov 03 01:18:16.767010 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:16.766986 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-htfx6" event={"ID":"766a044f-fc66-4998-8006-343c0329860d","Type":"ContainerStarted","Data":"a0a9ec2bdcdd88a98fc92b6fa9724aac0472c842079f5b018614d7d35bef61a2"} Nov 03 01:18:16.770168 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:16.770149 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lcwkf" event={"ID":"0e35c4cd-ef18-40c1-8568-f77773e378b2","Type":"ContainerStarted","Data":"bc2e9247f68ab8023446f16663f7ad019ff99fa6088f7afdb27af40b12373ab9"} Nov 03 01:18:16.770251 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:16.770195 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lcwkf" event={"ID":"0e35c4cd-ef18-40c1-8568-f77773e378b2","Type":"ContainerStarted","Data":"5a3286620029aafab039b5f2fad07f216c5b4cf21ba35eb27778eda606661e6c"} Nov 03 01:18:16.770251 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:16.770214 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lcwkf" event={"ID":"0e35c4cd-ef18-40c1-8568-f77773e378b2","Type":"ContainerStarted","Data":"a8e0819cc9cee4743891c1bd3a02cdedd551a431d84689742656f35cef6c34de"} Nov 03 01:18:16.883109 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:16.883056 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-4vr4n" podStartSLOduration=2.622879897 podStartE2EDuration="5.883042176s" totalImagesPullingTime="3.260162279s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:18:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:18:16.882861409 +0000 UTC m=+52.969716990" watchObservedRunningTime="2026-11-03 01:18:16.883042176 +0000 UTC m=+52.969897757" Nov 03 01:18:16.963017 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:16.962939 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:17.004053 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:17.004000 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-bbfdw" podStartSLOduration=2.059791106 podStartE2EDuration="6.003982018s" totalImagesPullingTime="3.944190912s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 01:18:11 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:18:17.003451546 +0000 UTC m=+53.090307128" watchObservedRunningTime="2026-11-03 01:18:17.003982018 +0000 UTC m=+53.090837603" Nov 03 01:18:17.173662 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:17.173628 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:17.777228 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:17.777146 2578 generic.go:356] "Generic (PLEG): container finished" podID="30eede54-1970-4f52-b219-70d3c469ddd4" containerID="569da36b752ded8ec2d1f5e3ab6261cebfb8cc955d1e71d8d2b15cf031367b62" exitCode=0 Nov 03 01:18:17.777655 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:17.777237 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5z5ct" event={"ID":"30eede54-1970-4f52-b219-70d3c469ddd4","Type":"ContainerDied","Data":"569da36b752ded8ec2d1f5e3ab6261cebfb8cc955d1e71d8d2b15cf031367b62"} Nov 03 01:18:17.780805 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:17.780777 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bbfdw" event={"ID":"2320cce9-0fdb-470a-ae01-e88118ba0419","Type":"ContainerStarted","Data":"5919db1a2763bae7517ea147405b13499cad4300dd093fdfd03fe40ff4eb914b"} Nov 03 01:18:17.851476 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:17.851439 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:18.154298 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:18.154271 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-8bbf59f7f-cmcdf"] Nov 03 01:18:18.156901 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:18.156877 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-8bbf59f7f-cmcdf" Nov 03 01:18:18.165794 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:18.165770 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8f478179-5984-48dd-b872-8f552954ecea-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-8bbf59f7f-cmcdf\" (UID: \"8f478179-5984-48dd-b872-8f552954ecea\") " pod="openshift-monitoring/thanos-querier-8bbf59f7f-cmcdf" Nov 03 01:18:18.165867 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:18.165822 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/8f478179-5984-48dd-b872-8f552954ecea-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-8bbf59f7f-cmcdf\" (UID: \"8f478179-5984-48dd-b872-8f552954ecea\") " pod="openshift-monitoring/thanos-querier-8bbf59f7f-cmcdf" Nov 03 01:18:18.165867 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:18.165843 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8l47\" (UniqueName: \"kubernetes.io/projected/8f478179-5984-48dd-b872-8f552954ecea-kube-api-access-v8l47\") pod \"thanos-querier-8bbf59f7f-cmcdf\" (UID: \"8f478179-5984-48dd-b872-8f552954ecea\") " pod="openshift-monitoring/thanos-querier-8bbf59f7f-cmcdf" Nov 03 01:18:18.165972 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:18.165870 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/8f478179-5984-48dd-b872-8f552954ecea-secret-thanos-querier-tls\") pod \"thanos-querier-8bbf59f7f-cmcdf\" (UID: \"8f478179-5984-48dd-b872-8f552954ecea\") " pod="openshift-monitoring/thanos-querier-8bbf59f7f-cmcdf" Nov 03 01:18:18.165972 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:18.165894 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8f478179-5984-48dd-b872-8f552954ecea-tmp\") pod \"thanos-querier-8bbf59f7f-cmcdf\" (UID: \"8f478179-5984-48dd-b872-8f552954ecea\") " pod="openshift-monitoring/thanos-querier-8bbf59f7f-cmcdf" Nov 03 01:18:18.165972 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:18.165930 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8f478179-5984-48dd-b872-8f552954ecea-secret-grpc-tls\") pod \"thanos-querier-8bbf59f7f-cmcdf\" (UID: \"8f478179-5984-48dd-b872-8f552954ecea\") " pod="openshift-monitoring/thanos-querier-8bbf59f7f-cmcdf" Nov 03 01:18:18.165972 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:18.165955 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/8f478179-5984-48dd-b872-8f552954ecea-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-8bbf59f7f-cmcdf\" (UID: \"8f478179-5984-48dd-b872-8f552954ecea\") " pod="openshift-monitoring/thanos-querier-8bbf59f7f-cmcdf" Nov 03 01:18:18.166166 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:18.165984 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8f478179-5984-48dd-b872-8f552954ecea-metrics-client-ca\") pod \"thanos-querier-8bbf59f7f-cmcdf\" (UID: \"8f478179-5984-48dd-b872-8f552954ecea\") " pod="openshift-monitoring/thanos-querier-8bbf59f7f-cmcdf" Nov 03 01:18:18.166166 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:18.166025 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8f478179-5984-48dd-b872-8f552954ecea-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-8bbf59f7f-cmcdf\" (UID: \"8f478179-5984-48dd-b872-8f552954ecea\") " pod="openshift-monitoring/thanos-querier-8bbf59f7f-cmcdf" Nov 03 01:18:18.217856 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:18.217750 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-8bbf59f7f-cmcdf"] Nov 03 01:18:18.267206 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:18.267173 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8f478179-5984-48dd-b872-8f552954ecea-metrics-client-ca\") pod \"thanos-querier-8bbf59f7f-cmcdf\" (UID: \"8f478179-5984-48dd-b872-8f552954ecea\") " pod="openshift-monitoring/thanos-querier-8bbf59f7f-cmcdf" Nov 03 01:18:18.267430 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:18.267239 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8f478179-5984-48dd-b872-8f552954ecea-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-8bbf59f7f-cmcdf\" (UID: \"8f478179-5984-48dd-b872-8f552954ecea\") " pod="openshift-monitoring/thanos-querier-8bbf59f7f-cmcdf" Nov 03 01:18:18.267430 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:18.267277 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8f478179-5984-48dd-b872-8f552954ecea-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-8bbf59f7f-cmcdf\" (UID: \"8f478179-5984-48dd-b872-8f552954ecea\") " pod="openshift-monitoring/thanos-querier-8bbf59f7f-cmcdf" Nov 03 01:18:18.267430 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:18.267313 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/8f478179-5984-48dd-b872-8f552954ecea-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-8bbf59f7f-cmcdf\" (UID: \"8f478179-5984-48dd-b872-8f552954ecea\") " pod="openshift-monitoring/thanos-querier-8bbf59f7f-cmcdf" Nov 03 01:18:18.267430 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:18.267341 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8l47\" (UniqueName: \"kubernetes.io/projected/8f478179-5984-48dd-b872-8f552954ecea-kube-api-access-v8l47\") pod \"thanos-querier-8bbf59f7f-cmcdf\" (UID: \"8f478179-5984-48dd-b872-8f552954ecea\") " pod="openshift-monitoring/thanos-querier-8bbf59f7f-cmcdf" Nov 03 01:18:18.267430 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:18.267387 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/8f478179-5984-48dd-b872-8f552954ecea-secret-thanos-querier-tls\") pod \"thanos-querier-8bbf59f7f-cmcdf\" (UID: \"8f478179-5984-48dd-b872-8f552954ecea\") " pod="openshift-monitoring/thanos-querier-8bbf59f7f-cmcdf" Nov 03 01:18:18.267687 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:18.267442 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8f478179-5984-48dd-b872-8f552954ecea-tmp\") pod \"thanos-querier-8bbf59f7f-cmcdf\" (UID: \"8f478179-5984-48dd-b872-8f552954ecea\") " pod="openshift-monitoring/thanos-querier-8bbf59f7f-cmcdf" Nov 03 01:18:18.267687 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:18.267498 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8f478179-5984-48dd-b872-8f552954ecea-secret-grpc-tls\") pod \"thanos-querier-8bbf59f7f-cmcdf\" (UID: \"8f478179-5984-48dd-b872-8f552954ecea\") " pod="openshift-monitoring/thanos-querier-8bbf59f7f-cmcdf" Nov 03 01:18:18.267687 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:18.267525 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/8f478179-5984-48dd-b872-8f552954ecea-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-8bbf59f7f-cmcdf\" (UID: \"8f478179-5984-48dd-b872-8f552954ecea\") " pod="openshift-monitoring/thanos-querier-8bbf59f7f-cmcdf" Nov 03 01:18:18.267876 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:18.267851 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8f478179-5984-48dd-b872-8f552954ecea-tmp\") pod \"thanos-querier-8bbf59f7f-cmcdf\" (UID: \"8f478179-5984-48dd-b872-8f552954ecea\") " pod="openshift-monitoring/thanos-querier-8bbf59f7f-cmcdf" Nov 03 01:18:18.268073 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:18.268049 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8f478179-5984-48dd-b872-8f552954ecea-metrics-client-ca\") pod \"thanos-querier-8bbf59f7f-cmcdf\" (UID: \"8f478179-5984-48dd-b872-8f552954ecea\") " pod="openshift-monitoring/thanos-querier-8bbf59f7f-cmcdf" Nov 03 01:18:18.291934 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:18.291905 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Nov 03 01:18:18.292097 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:18.292018 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Nov 03 01:18:18.292097 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:18.292075 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Nov 03 01:18:18.292258 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:18.292121 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Nov 03 01:18:18.292258 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:18.292136 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-rgvh9\"" Nov 03 01:18:18.292258 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:18.292179 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-8hlmd393unt3b\"" Nov 03 01:18:18.292258 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:18.292238 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Nov 03 01:18:18.301640 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:18.301605 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8f478179-5984-48dd-b872-8f552954ecea-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-8bbf59f7f-cmcdf\" (UID: \"8f478179-5984-48dd-b872-8f552954ecea\") " pod="openshift-monitoring/thanos-querier-8bbf59f7f-cmcdf" Nov 03 01:18:18.302132 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:18.302104 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8f478179-5984-48dd-b872-8f552954ecea-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-8bbf59f7f-cmcdf\" (UID: \"8f478179-5984-48dd-b872-8f552954ecea\") " pod="openshift-monitoring/thanos-querier-8bbf59f7f-cmcdf" Nov 03 01:18:18.302652 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:18.302618 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8f478179-5984-48dd-b872-8f552954ecea-secret-grpc-tls\") pod \"thanos-querier-8bbf59f7f-cmcdf\" (UID: \"8f478179-5984-48dd-b872-8f552954ecea\") " pod="openshift-monitoring/thanos-querier-8bbf59f7f-cmcdf" Nov 03 01:18:18.302798 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:18.302774 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/8f478179-5984-48dd-b872-8f552954ecea-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-8bbf59f7f-cmcdf\" (UID: \"8f478179-5984-48dd-b872-8f552954ecea\") " pod="openshift-monitoring/thanos-querier-8bbf59f7f-cmcdf" Nov 03 01:18:18.302850 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:18.302808 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/8f478179-5984-48dd-b872-8f552954ecea-secret-thanos-querier-tls\") pod \"thanos-querier-8bbf59f7f-cmcdf\" (UID: \"8f478179-5984-48dd-b872-8f552954ecea\") " pod="openshift-monitoring/thanos-querier-8bbf59f7f-cmcdf" Nov 03 01:18:18.303334 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:18.303314 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/8f478179-5984-48dd-b872-8f552954ecea-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-8bbf59f7f-cmcdf\" (UID: \"8f478179-5984-48dd-b872-8f552954ecea\") " pod="openshift-monitoring/thanos-querier-8bbf59f7f-cmcdf" Nov 03 01:18:18.348600 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:18.348524 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8l47\" (UniqueName: \"kubernetes.io/projected/8f478179-5984-48dd-b872-8f552954ecea-kube-api-access-v8l47\") pod \"thanos-querier-8bbf59f7f-cmcdf\" (UID: \"8f478179-5984-48dd-b872-8f552954ecea\") " pod="openshift-monitoring/thanos-querier-8bbf59f7f-cmcdf" Nov 03 01:18:18.475320 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:18.475285 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-8bbf59f7f-cmcdf" Nov 03 01:18:18.785781 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:18.785745 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lcwkf" event={"ID":"0e35c4cd-ef18-40c1-8568-f77773e378b2","Type":"ContainerStarted","Data":"4830d7b5a0fa43c2ed1f451e5a5f80eda995d9a9c7e92421036c6a2ebf0e2cc3"} Nov 03 01:18:18.787961 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:18.787936 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5z5ct" event={"ID":"30eede54-1970-4f52-b219-70d3c469ddd4","Type":"ContainerStarted","Data":"6a3620f14bec1342c4f02e1a5e9490309c4bb1fc7667695fd6f21b12c89eb7b2"} Nov 03 01:18:18.788090 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:18.787980 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5z5ct" event={"ID":"30eede54-1970-4f52-b219-70d3c469ddd4","Type":"ContainerStarted","Data":"3ab7ccca158ed055414d74b9ac1bafdb6a3499636f6edfe4318053d49e6ee606"} Nov 03 01:18:18.790063 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:18.790040 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-htfx6" event={"ID":"766a044f-fc66-4998-8006-343c0329860d","Type":"ContainerStarted","Data":"269d155ba68fb4eaca009d242233386c4b388475f95dec811afe551e8ec2530c"} Nov 03 01:18:18.790134 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:18.790084 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-htfx6" event={"ID":"766a044f-fc66-4998-8006-343c0329860d","Type":"ContainerStarted","Data":"2437a1a83b9205293761ab5000d84be417d9514212af916c552fc1562c7afd1e"} Nov 03 01:18:18.790134 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:18.790108 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-htfx6" event={"ID":"766a044f-fc66-4998-8006-343c0329860d","Type":"ContainerStarted","Data":"b708cc8c44f322ea9e31c2c4ee4ea167e82e011585fe1539651eeb733e2bfa45"} Nov 03 01:18:18.907943 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:18.907855 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lcwkf" podStartSLOduration=2.085285248 podStartE2EDuration="3.907841417s" totalImagesPullingTime="1.822556169s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:18:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:18:18.907074222 +0000 UTC m=+54.993929802" watchObservedRunningTime="2026-11-03 01:18:18.907841417 +0000 UTC m=+54.994696997" Nov 03 01:18:18.951981 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:18.951951 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-8bbf59f7f-cmcdf"] Nov 03 01:18:18.978884 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:18:18.978859 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f478179_5984_48dd_b872_8f552954ecea.slice/crio-4d1338fb1357347638c76068111fb2982b81469361736cb026758c7679f468a2 WatchSource:0}: Error finding container 4d1338fb1357347638c76068111fb2982b81469361736cb026758c7679f468a2: Status 404 returned error can't find the container with id 4d1338fb1357347638c76068111fb2982b81469361736cb026758c7679f468a2 Nov 03 01:18:19.031862 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:19.031809 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-htfx6" podStartSLOduration=1.779962775 podStartE2EDuration="4.031791263s" totalImagesPullingTime="2.251828488s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:18:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:18:19.031184755 +0000 UTC m=+55.118040336" watchObservedRunningTime="2026-11-03 01:18:19.031791263 +0000 UTC m=+55.118646845" Nov 03 01:18:19.156427 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:19.156351 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-5z5ct" podStartSLOduration=2.433304024 podStartE2EDuration="4.156336642s" totalImagesPullingTime="1.637182616s" totalInitContainerRuntime="85.850002ms" isStatefulPod=true podCreationTimestamp="2026-11-03 01:18:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:18:19.154181086 +0000 UTC m=+55.241036667" watchObservedRunningTime="2026-11-03 01:18:19.156336642 +0000 UTC m=+55.243192258" Nov 03 01:18:19.736545 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:19.736508 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:19.798473 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:19.798431 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8bbf59f7f-cmcdf" event={"ID":"8f478179-5984-48dd-b872-8f552954ecea","Type":"ContainerStarted","Data":"4d1338fb1357347638c76068111fb2982b81469361736cb026758c7679f468a2"} Nov 03 01:18:20.001655 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:20.001564 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:20.210521 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:20.210482 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:20.237151 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:20.237121 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-ssqxs"] Nov 03 01:18:20.239240 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:20.239213 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-ssqxs" Nov 03 01:18:20.285544 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:20.285476 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjdgx\" (UniqueName: \"kubernetes.io/projected/9eb34322-124e-484d-a4fa-d79391ac36b1-kube-api-access-zjdgx\") pod \"monitoring-plugin-77f4f76594-ssqxs\" (UID: \"9eb34322-124e-484d-a4fa-d79391ac36b1\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-ssqxs" Nov 03 01:18:20.285678 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:20.285558 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9eb34322-124e-484d-a4fa-d79391ac36b1-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-ssqxs\" (UID: \"9eb34322-124e-484d-a4fa-d79391ac36b1\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-ssqxs" Nov 03 01:18:20.295781 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:20.295739 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-bh2h6\"" Nov 03 01:18:20.295950 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:20.295863 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Nov 03 01:18:20.297021 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:20.296999 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-ssqxs"] Nov 03 01:18:20.386927 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:20.386892 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjdgx\" (UniqueName: \"kubernetes.io/projected/9eb34322-124e-484d-a4fa-d79391ac36b1-kube-api-access-zjdgx\") pod \"monitoring-plugin-77f4f76594-ssqxs\" (UID: \"9eb34322-124e-484d-a4fa-d79391ac36b1\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-ssqxs" Nov 03 01:18:20.387095 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:20.386957 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9eb34322-124e-484d-a4fa-d79391ac36b1-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-ssqxs\" (UID: \"9eb34322-124e-484d-a4fa-d79391ac36b1\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-ssqxs" Nov 03 01:18:20.389613 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:20.389589 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9eb34322-124e-484d-a4fa-d79391ac36b1-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-ssqxs\" (UID: \"9eb34322-124e-484d-a4fa-d79391ac36b1\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-ssqxs" Nov 03 01:18:20.421979 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:20.421951 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:20.450529 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:20.450500 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjdgx\" (UniqueName: \"kubernetes.io/projected/9eb34322-124e-484d-a4fa-d79391ac36b1-kube-api-access-zjdgx\") pod \"monitoring-plugin-77f4f76594-ssqxs\" (UID: \"9eb34322-124e-484d-a4fa-d79391ac36b1\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-ssqxs" Nov 03 01:18:20.551879 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:20.551844 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-ssqxs" Nov 03 01:18:20.653154 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:20.653123 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:20.925052 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:18:20.925016 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9eb34322_124e_484d_a4fa_d79391ac36b1.slice/crio-97a126238281ad8365de918d31b44613980842c7d124f305133855532128b7c3 WatchSource:0}: Error finding container 97a126238281ad8365de918d31b44613980842c7d124f305133855532128b7c3: Status 404 returned error can't find the container with id 97a126238281ad8365de918d31b44613980842c7d124f305133855532128b7c3 Nov 03 01:18:20.928347 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:20.928324 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-ssqxs"] Nov 03 01:18:21.806037 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:21.806000 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-ssqxs" event={"ID":"9eb34322-124e-484d-a4fa-d79391ac36b1","Type":"ContainerStarted","Data":"97a126238281ad8365de918d31b44613980842c7d124f305133855532128b7c3"} Nov 03 01:18:22.812901 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:22.812863 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8bbf59f7f-cmcdf" event={"ID":"8f478179-5984-48dd-b872-8f552954ecea","Type":"ContainerStarted","Data":"1c9b758bee4195e4d1c2ba58fef4ebaef26afc57ea9ef59b9887a566045b69ec"} Nov 03 01:18:22.813352 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:22.812949 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8bbf59f7f-cmcdf" event={"ID":"8f478179-5984-48dd-b872-8f552954ecea","Type":"ContainerStarted","Data":"f6cb7d0b97ce959365a920a7824b88bfb4b8e16efd172c44d193a4037844ed82"} Nov 03 01:18:22.813352 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:22.813006 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8bbf59f7f-cmcdf" event={"ID":"8f478179-5984-48dd-b872-8f552954ecea","Type":"ContainerStarted","Data":"55195a377d339c3ca1183fa6b6849a53e989f56c288f4ed4d538d79cc0c80279"} Nov 03 01:18:23.398481 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:23.398441 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-69fcb6c6b8-jqgxm"] Nov 03 01:18:23.401176 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:23.401151 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69fcb6c6b8-jqgxm" Nov 03 01:18:23.412466 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:23.412442 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/386397a9-5c8d-4221-894b-279e0b396afe-console-oauth-config\") pod \"console-69fcb6c6b8-jqgxm\" (UID: \"386397a9-5c8d-4221-894b-279e0b396afe\") " pod="openshift-console/console-69fcb6c6b8-jqgxm" Nov 03 01:18:23.412595 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:23.412485 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/386397a9-5c8d-4221-894b-279e0b396afe-oauth-serving-cert\") pod \"console-69fcb6c6b8-jqgxm\" (UID: \"386397a9-5c8d-4221-894b-279e0b396afe\") " pod="openshift-console/console-69fcb6c6b8-jqgxm" Nov 03 01:18:23.412595 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:23.412502 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/386397a9-5c8d-4221-894b-279e0b396afe-console-serving-cert\") pod \"console-69fcb6c6b8-jqgxm\" (UID: \"386397a9-5c8d-4221-894b-279e0b396afe\") " pod="openshift-console/console-69fcb6c6b8-jqgxm" Nov 03 01:18:23.412595 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:23.412564 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/386397a9-5c8d-4221-894b-279e0b396afe-trusted-ca-bundle\") pod \"console-69fcb6c6b8-jqgxm\" (UID: \"386397a9-5c8d-4221-894b-279e0b396afe\") " pod="openshift-console/console-69fcb6c6b8-jqgxm" Nov 03 01:18:23.412716 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:23.412667 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/386397a9-5c8d-4221-894b-279e0b396afe-tmp\") pod \"console-69fcb6c6b8-jqgxm\" (UID: \"386397a9-5c8d-4221-894b-279e0b396afe\") " pod="openshift-console/console-69fcb6c6b8-jqgxm" Nov 03 01:18:23.412716 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:23.412707 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/386397a9-5c8d-4221-894b-279e0b396afe-console-config\") pod \"console-69fcb6c6b8-jqgxm\" (UID: \"386397a9-5c8d-4221-894b-279e0b396afe\") " pod="openshift-console/console-69fcb6c6b8-jqgxm" Nov 03 01:18:23.412801 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:23.412732 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/386397a9-5c8d-4221-894b-279e0b396afe-service-ca\") pod \"console-69fcb6c6b8-jqgxm\" (UID: \"386397a9-5c8d-4221-894b-279e0b396afe\") " pod="openshift-console/console-69fcb6c6b8-jqgxm" Nov 03 01:18:23.412846 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:23.412803 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wk6dw\" (UniqueName: \"kubernetes.io/projected/386397a9-5c8d-4221-894b-279e0b396afe-kube-api-access-wk6dw\") pod \"console-69fcb6c6b8-jqgxm\" (UID: \"386397a9-5c8d-4221-894b-279e0b396afe\") " pod="openshift-console/console-69fcb6c6b8-jqgxm" Nov 03 01:18:23.457332 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:23.457305 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Nov 03 01:18:23.509108 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:23.509077 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-fc5zz\"" Nov 03 01:18:23.509290 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:23.509120 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Nov 03 01:18:23.509290 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:23.509124 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Nov 03 01:18:23.509290 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:23.509115 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Nov 03 01:18:23.509290 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:23.509161 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Nov 03 01:18:23.509290 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:23.509143 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Nov 03 01:18:23.509290 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:23.509127 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Nov 03 01:18:23.513681 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:23.513662 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/386397a9-5c8d-4221-894b-279e0b396afe-console-oauth-config\") pod \"console-69fcb6c6b8-jqgxm\" (UID: \"386397a9-5c8d-4221-894b-279e0b396afe\") " pod="openshift-console/console-69fcb6c6b8-jqgxm" Nov 03 01:18:23.513768 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:23.513711 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/386397a9-5c8d-4221-894b-279e0b396afe-oauth-serving-cert\") pod \"console-69fcb6c6b8-jqgxm\" (UID: \"386397a9-5c8d-4221-894b-279e0b396afe\") " pod="openshift-console/console-69fcb6c6b8-jqgxm" Nov 03 01:18:23.513768 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:23.513735 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/386397a9-5c8d-4221-894b-279e0b396afe-console-serving-cert\") pod \"console-69fcb6c6b8-jqgxm\" (UID: \"386397a9-5c8d-4221-894b-279e0b396afe\") " pod="openshift-console/console-69fcb6c6b8-jqgxm" Nov 03 01:18:23.513768 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:23.513763 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/386397a9-5c8d-4221-894b-279e0b396afe-trusted-ca-bundle\") pod \"console-69fcb6c6b8-jqgxm\" (UID: \"386397a9-5c8d-4221-894b-279e0b396afe\") " pod="openshift-console/console-69fcb6c6b8-jqgxm" Nov 03 01:18:23.513925 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:23.513800 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/386397a9-5c8d-4221-894b-279e0b396afe-tmp\") pod \"console-69fcb6c6b8-jqgxm\" (UID: \"386397a9-5c8d-4221-894b-279e0b396afe\") " pod="openshift-console/console-69fcb6c6b8-jqgxm" Nov 03 01:18:23.513925 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:23.513848 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/386397a9-5c8d-4221-894b-279e0b396afe-console-config\") pod \"console-69fcb6c6b8-jqgxm\" (UID: \"386397a9-5c8d-4221-894b-279e0b396afe\") " pod="openshift-console/console-69fcb6c6b8-jqgxm" Nov 03 01:18:23.513925 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:23.513888 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/386397a9-5c8d-4221-894b-279e0b396afe-service-ca\") pod \"console-69fcb6c6b8-jqgxm\" (UID: \"386397a9-5c8d-4221-894b-279e0b396afe\") " pod="openshift-console/console-69fcb6c6b8-jqgxm" Nov 03 01:18:23.514063 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:23.513922 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wk6dw\" (UniqueName: \"kubernetes.io/projected/386397a9-5c8d-4221-894b-279e0b396afe-kube-api-access-wk6dw\") pod \"console-69fcb6c6b8-jqgxm\" (UID: \"386397a9-5c8d-4221-894b-279e0b396afe\") " pod="openshift-console/console-69fcb6c6b8-jqgxm" Nov 03 01:18:23.514261 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:23.514239 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/386397a9-5c8d-4221-894b-279e0b396afe-tmp\") pod \"console-69fcb6c6b8-jqgxm\" (UID: \"386397a9-5c8d-4221-894b-279e0b396afe\") " pod="openshift-console/console-69fcb6c6b8-jqgxm" Nov 03 01:18:23.514569 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:23.514551 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/386397a9-5c8d-4221-894b-279e0b396afe-console-config\") pod \"console-69fcb6c6b8-jqgxm\" (UID: \"386397a9-5c8d-4221-894b-279e0b396afe\") " pod="openshift-console/console-69fcb6c6b8-jqgxm" Nov 03 01:18:23.514624 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:23.514562 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/386397a9-5c8d-4221-894b-279e0b396afe-oauth-serving-cert\") pod \"console-69fcb6c6b8-jqgxm\" (UID: \"386397a9-5c8d-4221-894b-279e0b396afe\") " pod="openshift-console/console-69fcb6c6b8-jqgxm" Nov 03 01:18:23.514624 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:23.514613 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/386397a9-5c8d-4221-894b-279e0b396afe-service-ca\") pod \"console-69fcb6c6b8-jqgxm\" (UID: \"386397a9-5c8d-4221-894b-279e0b396afe\") " pod="openshift-console/console-69fcb6c6b8-jqgxm" Nov 03 01:18:23.516594 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:23.516578 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/386397a9-5c8d-4221-894b-279e0b396afe-console-oauth-config\") pod \"console-69fcb6c6b8-jqgxm\" (UID: \"386397a9-5c8d-4221-894b-279e0b396afe\") " pod="openshift-console/console-69fcb6c6b8-jqgxm" Nov 03 01:18:23.516772 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:23.516753 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/386397a9-5c8d-4221-894b-279e0b396afe-console-serving-cert\") pod \"console-69fcb6c6b8-jqgxm\" (UID: \"386397a9-5c8d-4221-894b-279e0b396afe\") " pod="openshift-console/console-69fcb6c6b8-jqgxm" Nov 03 01:18:23.613560 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:23.613528 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Nov 03 01:18:23.614660 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:23.614636 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-69fcb6c6b8-jqgxm"] Nov 03 01:18:23.614660 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:23.614653 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/386397a9-5c8d-4221-894b-279e0b396afe-trusted-ca-bundle\") pod \"console-69fcb6c6b8-jqgxm\" (UID: \"386397a9-5c8d-4221-894b-279e0b396afe\") " pod="openshift-console/console-69fcb6c6b8-jqgxm" Nov 03 01:18:23.615996 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:23.615973 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wk6dw\" (UniqueName: \"kubernetes.io/projected/386397a9-5c8d-4221-894b-279e0b396afe-kube-api-access-wk6dw\") pod \"console-69fcb6c6b8-jqgxm\" (UID: \"386397a9-5c8d-4221-894b-279e0b396afe\") " pod="openshift-console/console-69fcb6c6b8-jqgxm" Nov 03 01:18:23.711577 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:23.711496 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69fcb6c6b8-jqgxm" Nov 03 01:18:23.817909 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:23.817846 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-ssqxs" event={"ID":"9eb34322-124e-484d-a4fa-d79391ac36b1","Type":"ContainerStarted","Data":"4cf9adc4f62c6b38944b5e392fb1674c0385f22034d55590238219920fbfb8db"} Nov 03 01:18:23.818356 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:23.818079 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-ssqxs" Nov 03 01:18:23.822719 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:23.822698 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-ssqxs" Nov 03 01:18:23.950652 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:23.950599 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-ssqxs" podStartSLOduration=2.004787534 podStartE2EDuration="3.950584751s" totalImagesPullingTime="1.945797217s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:18:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:18:23.950540947 +0000 UTC m=+60.037396531" watchObservedRunningTime="2026-11-03 01:18:23.950584751 +0000 UTC m=+60.037440346" Nov 03 01:18:24.088061 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:24.088032 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-69fcb6c6b8-jqgxm"] Nov 03 01:18:24.682043 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:24.682009 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:24.716437 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:24.716409 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-n2cjg" Nov 03 01:18:24.824024 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:24.823980 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8bbf59f7f-cmcdf" event={"ID":"8f478179-5984-48dd-b872-8f552954ecea","Type":"ContainerStarted","Data":"821069eee2ea12ec6ea28a7f3725c3dd566de8c8828cf3d9e384cd39dbb5bfeb"} Nov 03 01:18:24.824483 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:24.824061 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8bbf59f7f-cmcdf" event={"ID":"8f478179-5984-48dd-b872-8f552954ecea","Type":"ContainerStarted","Data":"e2458246e609feb342a6434ba9849e7e472e6bacfd46c519bb7bb90933908b1e"} Nov 03 01:18:24.824483 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:24.824106 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8bbf59f7f-cmcdf" event={"ID":"8f478179-5984-48dd-b872-8f552954ecea","Type":"ContainerStarted","Data":"2327cc0e560c936ac66ea3baa7b35d696c33f0766594c3866a559e1b459d23be"} Nov 03 01:18:24.824483 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:24.824144 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-8bbf59f7f-cmcdf" Nov 03 01:18:24.825391 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:24.825349 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69fcb6c6b8-jqgxm" event={"ID":"386397a9-5c8d-4221-894b-279e0b396afe","Type":"ContainerStarted","Data":"7e4535033e2187d3a887d160f43211e0b3f012c6b23c0b04d27c966b04e03c50"} Nov 03 01:18:24.960734 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:24.960655 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:24.983961 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:24.983909 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-8bbf59f7f-cmcdf" podStartSLOduration=2.154255661 podStartE2EDuration="6.983896991s" totalImagesPullingTime="4.82964133s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:18:18 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:18:24.982855537 +0000 UTC m=+61.069711117" watchObservedRunningTime="2026-11-03 01:18:24.983896991 +0000 UTC m=+61.070752572" Nov 03 01:18:25.221416 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:25.221320 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:26.294301 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:26.294267 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:26.495673 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:26.495641 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:26.783626 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:26.783595 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-kmpbz" Nov 03 01:18:27.174134 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:27.174058 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:27.374287 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:27.374254 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:27.955556 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:27.955513 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:28.158102 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:28.158062 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:28.840621 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:28.840580 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69fcb6c6b8-jqgxm" event={"ID":"386397a9-5c8d-4221-894b-279e0b396afe","Type":"ContainerStarted","Data":"9a195ef36f598769559baef0a3d59ff9ce64c2296d588760a3562463e5240eb3"} Nov 03 01:18:28.962711 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:28.962658 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-69fcb6c6b8-jqgxm" podStartSLOduration=1.6409623 podStartE2EDuration="5.962643258s" totalImagesPullingTime="4.321680958s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:18:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:18:28.962152375 +0000 UTC m=+65.049007957" watchObservedRunningTime="2026-11-03 01:18:28.962643258 +0000 UTC m=+65.049498838" Nov 03 01:18:30.273978 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:30.273941 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6179266f-07ff-48cd-bf03-3f1e961529d7-original-pull-secret\") pod \"global-pull-secret-syncer-dfr9t\" (UID: \"6179266f-07ff-48cd-bf03-3f1e961529d7\") " pod="kube-system/global-pull-secret-syncer-dfr9t" Nov 03 01:18:30.274341 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:30.273992 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ecc2615-b458-41f8-839f-e22c53e7ad0e-metrics-certs\") pod \"network-metrics-daemon-8g56n\" (UID: \"6ecc2615-b458-41f8-839f-e22c53e7ad0e\") " pod="openshift-multus/network-metrics-daemon-8g56n" Nov 03 01:18:30.327295 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:30.327268 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 01:18:30.336496 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:30.336473 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ecc2615-b458-41f8-839f-e22c53e7ad0e-metrics-certs\") pod \"network-metrics-daemon-8g56n\" (UID: \"6ecc2615-b458-41f8-839f-e22c53e7ad0e\") " pod="openshift-multus/network-metrics-daemon-8g56n" Nov 03 01:18:30.375115 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:30.375079 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgd8w\" (UniqueName: \"kubernetes.io/projected/3b2bf2f6-28aa-4480-9f4a-f4e7f659e966-kube-api-access-vgd8w\") pod \"network-check-target-kg99x\" (UID: \"3b2bf2f6-28aa-4480-9f4a-f4e7f659e966\") " pod="openshift-network-diagnostics/network-check-target-kg99x" Nov 03 01:18:30.378938 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:30.378920 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 01:18:30.387081 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:30.387054 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6179266f-07ff-48cd-bf03-3f1e961529d7-original-pull-secret\") pod \"global-pull-secret-syncer-dfr9t\" (UID: \"6179266f-07ff-48cd-bf03-3f1e961529d7\") " pod="kube-system/global-pull-secret-syncer-dfr9t" Nov 03 01:18:30.428415 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:30.428387 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 01:18:30.488555 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:30.488526 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 01:18:30.498227 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:30.498197 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgd8w\" (UniqueName: \"kubernetes.io/projected/3b2bf2f6-28aa-4480-9f4a-f4e7f659e966-kube-api-access-vgd8w\") pod \"network-check-target-kg99x\" (UID: \"3b2bf2f6-28aa-4480-9f4a-f4e7f659e966\") " pod="openshift-network-diagnostics/network-check-target-kg99x" Nov 03 01:18:30.576692 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:30.576659 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-7sd2v\"" Nov 03 01:18:30.580664 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:30.580646 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-mmkcs\"" Nov 03 01:18:30.583830 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:30.583813 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8g56n" Nov 03 01:18:30.584797 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:30.584782 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-fqzf6\"" Nov 03 01:18:30.588579 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:30.588563 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kg99x" Nov 03 01:18:30.591804 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:30.591786 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dfr9t" Nov 03 01:18:30.834683 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:30.834607 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-8bbf59f7f-cmcdf" Nov 03 01:18:30.955262 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:30.955227 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-8g56n"] Nov 03 01:18:30.987904 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:18:30.987873 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ecc2615_b458_41f8_839f_e22c53e7ad0e.slice/crio-ce3ef08fafa3dfc7c74905d661a123ef822c6af06a08d3f565ed2340bed81e4a WatchSource:0}: Error finding container ce3ef08fafa3dfc7c74905d661a123ef822c6af06a08d3f565ed2340bed81e4a: Status 404 returned error can't find the container with id ce3ef08fafa3dfc7c74905d661a123ef822c6af06a08d3f565ed2340bed81e4a Nov 03 01:18:30.998605 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:18:30.998581 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b2bf2f6_28aa_4480_9f4a_f4e7f659e966.slice/crio-1e4ab311b886828acac64bcbb2ec9807a19f1a8f64a2faaf1ba578edef09bfd1 WatchSource:0}: Error finding container 1e4ab311b886828acac64bcbb2ec9807a19f1a8f64a2faaf1ba578edef09bfd1: Status 404 returned error can't find the container with id 1e4ab311b886828acac64bcbb2ec9807a19f1a8f64a2faaf1ba578edef09bfd1 Nov 03 01:18:31.008162 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:31.008140 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-kg99x"] Nov 03 01:18:31.008953 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:31.008928 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-dfr9t"] Nov 03 01:18:31.035223 ip-10-0-136-16 kubenswrapper[2578]: W1103 01:18:31.035194 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6179266f_07ff_48cd_bf03_3f1e961529d7.slice/crio-3718cadbf973e67ccd8645ba1efff38c7704ba18c1a9ca4d2d3c853a7544a95d WatchSource:0}: Error finding container 3718cadbf973e67ccd8645ba1efff38c7704ba18c1a9ca4d2d3c853a7544a95d: Status 404 returned error can't find the container with id 3718cadbf973e67ccd8645ba1efff38c7704ba18c1a9ca4d2d3c853a7544a95d Nov 03 01:18:31.851439 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:31.851395 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-kg99x" event={"ID":"3b2bf2f6-28aa-4480-9f4a-f4e7f659e966","Type":"ContainerStarted","Data":"1e4ab311b886828acac64bcbb2ec9807a19f1a8f64a2faaf1ba578edef09bfd1"} Nov 03 01:18:31.852408 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:31.852387 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8g56n" event={"ID":"6ecc2615-b458-41f8-839f-e22c53e7ad0e","Type":"ContainerStarted","Data":"ce3ef08fafa3dfc7c74905d661a123ef822c6af06a08d3f565ed2340bed81e4a"} Nov 03 01:18:31.853326 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:31.853305 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-dfr9t" event={"ID":"6179266f-07ff-48cd-bf03-3f1e961529d7","Type":"ContainerStarted","Data":"3718cadbf973e67ccd8645ba1efff38c7704ba18c1a9ca4d2d3c853a7544a95d"} Nov 03 01:18:32.860194 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:32.860117 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8g56n" event={"ID":"6ecc2615-b458-41f8-839f-e22c53e7ad0e","Type":"ContainerStarted","Data":"60e79da67174aa87f90682445ab7b9b29c81279ad2dad7b712e3c52103720b31"} Nov 03 01:18:33.301885 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:33.301851 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:33.504087 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:33.504041 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:33.711971 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:33.711892 2578 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-69fcb6c6b8-jqgxm" Nov 03 01:18:33.712127 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:33.711975 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-69fcb6c6b8-jqgxm" Nov 03 01:18:33.717171 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:33.717146 2578 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-69fcb6c6b8-jqgxm" Nov 03 01:18:33.866820 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:33.866447 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8g56n" event={"ID":"6ecc2615-b458-41f8-839f-e22c53e7ad0e","Type":"ContainerStarted","Data":"75439fa46d4381a414ef50f225d7bdff5281db82e1e496706c3024b7b602b119"} Nov 03 01:18:33.871796 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:33.871773 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-69fcb6c6b8-jqgxm" Nov 03 01:18:33.985627 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:33.985524 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-8g56n" podStartSLOduration=67.241175797 podStartE2EDuration="1m8.985504459s" totalImagesPullingTime="1.744328662s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:17:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:18:33.984822144 +0000 UTC m=+70.071677754" watchObservedRunningTime="2026-11-03 01:18:33.985504459 +0000 UTC m=+70.072360043" Nov 03 01:18:34.688752 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:34.688719 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:34.876660 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:34.876624 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-kg99x" event={"ID":"3b2bf2f6-28aa-4480-9f4a-f4e7f659e966","Type":"ContainerStarted","Data":"b31cc63bc8925920d96f59e4818993ef36250d93c2f8c3001bd5a0fa8b754a00"} Nov 03 01:18:34.913425 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:34.913388 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:34.995240 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:34.995140 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-kg99x" podStartSLOduration=66.199762787 podStartE2EDuration="1m9.995125731s" totalImagesPullingTime="3.795362944s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:17:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:18:34.994683923 +0000 UTC m=+71.081539505" watchObservedRunningTime="2026-11-03 01:18:34.995125731 +0000 UTC m=+71.081981312" Nov 03 01:18:35.296948 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:35.296922 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:35.501007 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:35.500968 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:35.726269 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:35.726191 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:35.880331 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:35.880301 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-kg99x" Nov 03 01:18:37.889680 ip-10-0-136-16 kubenswrapper[2578]: I1103 01:18:37.889646 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-dfr9t" event={"ID":"6179266f-07ff-48cd-bf03-3f1e961529d7","Type":"ContainerStarted","Data":"fb38254fe916c0f1e83bc0433c99ee613740d77c833c9ac7ac53a19e224de5dd"} Nov 03 01:18:37.906670 ip-10-0-136-16 systemd[1]: Stopping Kubernetes Kubelet... Nov 03 01:18:37.925808 ip-10-0-136-16 systemd[1]: kubelet.service: Deactivated successfully. Nov 03 01:18:37.926032 ip-10-0-136-16 systemd[1]: Stopped Kubernetes Kubelet. Nov 03 01:18:37.926094 ip-10-0-136-16 systemd[1]: kubelet.service: Consumed 4.219s CPU time. Nov 03 01:18:37.937907 ip-10-0-136-16 systemd[1]: Starting Kubernetes Kubelet... Nov 03 01:18:37.999634 ip-10-0-136-16 kubenswrapper[5437]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 01:18:37.999634 ip-10-0-136-16 kubenswrapper[5437]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 03 01:18:37.999634 ip-10-0-136-16 kubenswrapper[5437]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 01:18:37.999634 ip-10-0-136-16 kubenswrapper[5437]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 01:18:38.001640 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.001558 5437 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 01:18:38.001640 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.001639 5437 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 01:18:38.001640 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.001643 5437 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 01:18:38.001738 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.001646 5437 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 01:18:38.001738 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.001651 5437 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 01:18:38.001738 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.001653 5437 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 01:18:38.001738 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.001657 5437 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 01:18:38.001738 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.001660 5437 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 01:18:38.001738 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.001662 5437 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 01:18:38.001738 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.001665 5437 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 01:18:38.001738 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.001668 5437 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 01:18:38.001738 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.001671 5437 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 01:18:38.001738 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.001674 5437 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 01:18:38.001738 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.001677 5437 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 01:18:38.001738 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.001679 5437 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 01:18:38.001738 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.001682 5437 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 01:18:38.001738 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.001684 5437 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 01:18:38.001738 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.001688 5437 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 01:18:38.001738 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.001690 5437 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 01:18:38.001738 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.001693 5437 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 01:18:38.001738 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.001696 5437 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 01:18:38.001738 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.001698 5437 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 01:18:38.001738 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.001701 5437 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 01:18:38.002215 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.001703 5437 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 01:18:38.002215 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.001707 5437 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 01:18:38.002215 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.001710 5437 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 01:18:38.002215 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.001712 5437 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 01:18:38.002215 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.001715 5437 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 01:18:38.002215 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.001721 5437 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 01:18:38.002215 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.001724 5437 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 01:18:38.002215 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.001727 5437 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 01:18:38.002215 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.001730 5437 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 01:18:38.002215 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.001735 5437 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 01:18:38.002215 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.001738 5437 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 01:18:38.002215 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.001741 5437 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 01:18:38.002215 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.001745 5437 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 01:18:38.002215 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.001748 5437 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 01:18:38.002215 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.001751 5437 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 01:18:38.002215 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.001753 5437 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 01:18:38.002215 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.001756 5437 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 01:18:38.002215 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.001759 5437 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 01:18:38.002215 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.001762 5437 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 01:18:38.002731 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.001764 5437 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 01:18:38.002731 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.001767 5437 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 01:18:38.002731 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.001770 5437 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 01:18:38.002731 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.001772 5437 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 01:18:38.002731 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.001775 5437 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 01:18:38.002731 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.001780 5437 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 01:18:38.002731 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.001783 5437 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 01:18:38.002731 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.001787 5437 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 01:18:38.002731 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.001790 5437 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 01:18:38.002731 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.001793 5437 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 01:18:38.002731 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.001796 5437 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 01:18:38.002731 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.001799 5437 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 01:18:38.002731 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.001801 5437 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 01:18:38.002731 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.001804 5437 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 01:18:38.002731 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.001807 5437 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 01:18:38.002731 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.001810 5437 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 01:18:38.002731 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.001812 5437 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 01:18:38.002731 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.001815 5437 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 01:18:38.002731 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.001818 5437 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 01:18:38.003190 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.001821 5437 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 01:18:38.003190 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.001824 5437 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 01:18:38.003190 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.001826 5437 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 01:18:38.003190 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.001829 5437 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 01:18:38.003190 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.001832 5437 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 01:18:38.003190 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.001834 5437 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 01:18:38.003190 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.001837 5437 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 01:18:38.003190 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.001839 5437 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 01:18:38.003190 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.001842 5437 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 01:18:38.003190 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.001844 5437 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 01:18:38.003190 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.001847 5437 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 01:18:38.003190 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.001850 5437 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 01:18:38.003190 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.001852 5437 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 01:18:38.003190 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.001855 5437 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 01:18:38.003190 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.001857 5437 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 01:18:38.003190 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.001860 5437 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 01:18:38.003190 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.001862 5437 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 01:18:38.003190 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.001865 5437 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 01:18:38.003190 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.001867 5437 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 01:18:38.003673 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.001870 5437 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 01:18:38.003673 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.001873 5437 feature_gate.go:386] unrecognized feature gate: Example Nov 03 01:18:38.003673 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.001876 5437 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 01:18:38.003673 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.001881 5437 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 01:18:38.003673 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.001884 5437 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 01:18:38.003673 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.001886 5437 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 01:18:38.003673 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.001889 5437 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 01:18:38.003673 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.001891 5437 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 01:18:38.003673 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.001895 5437 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 01:18:38.003673 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.001898 5437 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 01:18:38.003673 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.001900 5437 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 01:18:38.003673 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.001903 5437 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 01:18:38.003673 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.001906 5437 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 01:18:38.003673 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.001908 5437 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 01:18:38.003673 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.001911 5437 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 01:18:38.003673 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.001914 5437 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 01:18:38.003673 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.001916 5437 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 01:18:38.003673 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.001919 5437 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 01:18:38.003673 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.001922 5437 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 01:18:38.003673 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.001924 5437 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 01:18:38.004161 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.001927 5437 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 01:18:38.004161 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.001929 5437 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 01:18:38.004161 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.001932 5437 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 01:18:38.004161 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.001936 5437 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 01:18:38.004161 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.001938 5437 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 01:18:38.004161 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.001941 5437 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 01:18:38.004161 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.002746 5437 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 01:18:38.004161 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.002754 5437 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 01:18:38.004161 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.002757 5437 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 01:18:38.004161 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.002760 5437 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 01:18:38.004161 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.002763 5437 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 01:18:38.004161 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.002766 5437 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 01:18:38.004161 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.002768 5437 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 01:18:38.004161 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.002771 5437 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 01:18:38.004161 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.002774 5437 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 01:18:38.004161 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.002779 5437 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 01:18:38.004161 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.002781 5437 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 01:18:38.004161 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.002784 5437 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 01:18:38.004161 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.002787 5437 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 01:18:38.004161 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.002789 5437 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 01:18:38.004675 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.002792 5437 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 01:18:38.004675 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.002795 5437 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 01:18:38.004675 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.002798 5437 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 01:18:38.004675 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.002801 5437 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 01:18:38.004675 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.002803 5437 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 01:18:38.004675 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.002806 5437 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 01:18:38.004675 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.002809 5437 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 01:18:38.004675 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.002812 5437 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 01:18:38.004675 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.002815 5437 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 01:18:38.004675 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.002819 5437 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 01:18:38.004675 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.002823 5437 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 01:18:38.004675 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.002827 5437 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 01:18:38.004675 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.002831 5437 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 01:18:38.004675 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.002834 5437 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 01:18:38.004675 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.002839 5437 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 01:18:38.004675 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.002843 5437 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 01:18:38.004675 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.002847 5437 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 01:18:38.004675 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.002852 5437 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 01:18:38.004675 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.002856 5437 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 01:18:38.005150 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.002860 5437 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 01:18:38.005150 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.002863 5437 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 01:18:38.005150 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.002867 5437 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 01:18:38.005150 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.002871 5437 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 01:18:38.005150 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.002874 5437 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 01:18:38.005150 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.002877 5437 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 01:18:38.005150 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.002880 5437 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 01:18:38.005150 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.002882 5437 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 01:18:38.005150 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.002886 5437 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 01:18:38.005150 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.002889 5437 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 01:18:38.005150 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.002892 5437 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 01:18:38.005150 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.002895 5437 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 01:18:38.005150 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.002897 5437 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 01:18:38.005150 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.002900 5437 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 01:18:38.005150 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.002903 5437 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 01:18:38.005150 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.002906 5437 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 01:18:38.005150 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.002908 5437 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 01:18:38.005150 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.002911 5437 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 01:18:38.005150 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.002914 5437 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 01:18:38.005150 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.002917 5437 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 01:18:38.005649 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.002919 5437 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 01:18:38.005649 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.002923 5437 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 01:18:38.005649 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.002925 5437 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 01:18:38.005649 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.002928 5437 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 01:18:38.005649 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.002931 5437 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 01:18:38.005649 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.002933 5437 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 01:18:38.005649 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.002936 5437 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 01:18:38.005649 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.002938 5437 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 01:18:38.005649 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.002941 5437 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 01:18:38.005649 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.002943 5437 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 01:18:38.005649 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.002946 5437 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 01:18:38.005649 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.002949 5437 feature_gate.go:386] unrecognized feature gate: Example Nov 03 01:18:38.005649 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.002951 5437 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 01:18:38.005649 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.002955 5437 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 01:18:38.005649 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.002957 5437 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 01:18:38.005649 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.002960 5437 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 01:18:38.005649 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.002963 5437 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 01:18:38.005649 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.002965 5437 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 01:18:38.005649 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.002968 5437 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 01:18:38.005649 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.002970 5437 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 01:18:38.006131 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.002975 5437 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 01:18:38.006131 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.002978 5437 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 01:18:38.006131 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.002981 5437 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 01:18:38.006131 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.002983 5437 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 01:18:38.006131 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.002986 5437 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 01:18:38.006131 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.002989 5437 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 01:18:38.006131 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.002992 5437 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 01:18:38.006131 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.002995 5437 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 01:18:38.006131 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.002998 5437 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 01:18:38.006131 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.003000 5437 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 01:18:38.006131 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.003003 5437 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 01:18:38.006131 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.003006 5437 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 01:18:38.006131 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.003009 5437 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 01:18:38.006131 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.003012 5437 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 01:18:38.006131 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.003015 5437 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 01:18:38.006131 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.003017 5437 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 01:18:38.006131 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.003020 5437 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 01:18:38.006131 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.003023 5437 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 01:18:38.006131 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.003025 5437 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 01:18:38.006612 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.003028 5437 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 01:18:38.006612 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.003030 5437 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 01:18:38.006612 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.003033 5437 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 01:18:38.006612 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.003035 5437 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 01:18:38.006612 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.003038 5437 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 01:18:38.006612 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.003041 5437 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 01:18:38.006612 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.003043 5437 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 01:18:38.006612 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.003046 5437 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 01:18:38.006612 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.003049 5437 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 01:18:38.006612 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.003051 5437 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 01:18:38.006612 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.003054 5437 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 01:18:38.006612 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.003057 5437 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 01:18:38.006612 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.003060 5437 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 01:18:38.006612 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.003064 5437 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 01:18:38.006612 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.003571 5437 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 01:18:38.006612 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.003578 5437 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 01:18:38.006612 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.003582 5437 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 01:18:38.006612 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.003586 5437 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 01:18:38.006612 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.003591 5437 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 01:18:38.007090 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.003594 5437 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 01:18:38.007090 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.003597 5437 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 01:18:38.007090 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.003600 5437 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 01:18:38.007090 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.003602 5437 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 01:18:38.007090 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.003605 5437 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 01:18:38.007090 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.003608 5437 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 01:18:38.007090 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.003610 5437 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 01:18:38.007090 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.003613 5437 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 01:18:38.007090 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.003616 5437 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 01:18:38.007090 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.003619 5437 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 01:18:38.007090 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.003622 5437 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 01:18:38.007090 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.003624 5437 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 01:18:38.007090 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.003627 5437 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 01:18:38.007090 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.003630 5437 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 01:18:38.007090 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.003634 5437 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 01:18:38.007090 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.003637 5437 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 01:18:38.007090 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.003640 5437 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 01:18:38.007090 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.003642 5437 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 01:18:38.007090 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.003645 5437 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 01:18:38.007575 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.003648 5437 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 01:18:38.007575 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.003650 5437 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 01:18:38.007575 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.003653 5437 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 01:18:38.007575 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.003656 5437 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 01:18:38.007575 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.003658 5437 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 01:18:38.007575 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.003661 5437 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 01:18:38.007575 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.003663 5437 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 01:18:38.007575 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.003668 5437 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 01:18:38.007575 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.003670 5437 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 01:18:38.007575 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.003673 5437 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 01:18:38.007575 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.003676 5437 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 01:18:38.007575 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.003679 5437 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 01:18:38.007575 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.003683 5437 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 01:18:38.007575 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.003686 5437 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 01:18:38.007575 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.003688 5437 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 01:18:38.007575 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.003691 5437 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 01:18:38.007575 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.003694 5437 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 01:18:38.007575 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.003697 5437 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 01:18:38.007575 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.003700 5437 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 01:18:38.007575 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.003702 5437 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 01:18:38.008242 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.003705 5437 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 01:18:38.008242 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.003708 5437 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 01:18:38.008242 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.003711 5437 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 01:18:38.008242 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.003714 5437 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 01:18:38.008242 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.003716 5437 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 01:18:38.008242 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.003719 5437 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 01:18:38.008242 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.003721 5437 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 01:18:38.008242 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.003724 5437 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 01:18:38.008242 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.003726 5437 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 01:18:38.008242 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.003729 5437 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 01:18:38.008242 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.003731 5437 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 01:18:38.008242 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.003734 5437 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 01:18:38.008242 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.003736 5437 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 01:18:38.008242 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.003739 5437 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 01:18:38.008242 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.003742 5437 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 01:18:38.008242 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.003745 5437 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 01:18:38.008242 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.003747 5437 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 01:18:38.008242 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.003750 5437 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 01:18:38.008242 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.003754 5437 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 01:18:38.008973 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.003759 5437 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 01:18:38.008973 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.003762 5437 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 01:18:38.008973 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.003765 5437 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 01:18:38.008973 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.003767 5437 feature_gate.go:386] unrecognized feature gate: Example Nov 03 01:18:38.008973 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.003770 5437 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 01:18:38.008973 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.003777 5437 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 01:18:38.008973 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.003780 5437 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 01:18:38.008973 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.003782 5437 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 01:18:38.008973 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.003785 5437 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 01:18:38.008973 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.003788 5437 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 01:18:38.008973 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.003790 5437 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 01:18:38.008973 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.003793 5437 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 01:18:38.008973 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.003796 5437 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 01:18:38.008973 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.003798 5437 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 01:18:38.008973 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.003800 5437 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 01:18:38.008973 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.003803 5437 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 01:18:38.008973 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.003807 5437 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 01:18:38.008973 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.003809 5437 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 01:18:38.008973 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.003812 5437 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 01:18:38.008973 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.003815 5437 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 01:18:38.009482 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.003817 5437 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 01:18:38.009482 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.003820 5437 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 01:18:38.009482 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.003822 5437 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 01:18:38.009482 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.003825 5437 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 01:18:38.009482 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.003827 5437 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 01:18:38.009482 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.003830 5437 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 01:18:38.009482 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.003832 5437 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 01:18:38.009482 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.003835 5437 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 01:18:38.009482 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.003838 5437 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 01:18:38.009482 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.003840 5437 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 01:18:38.009482 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.003843 5437 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 01:18:38.009482 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.003846 5437 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 01:18:38.009482 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.003850 5437 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 01:18:38.009482 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.003853 5437 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 01:18:38.009482 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.003855 5437 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 01:18:38.009482 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.003858 5437 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 01:18:38.009482 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.003861 5437 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 01:18:38.009482 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.003864 5437 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 01:18:38.009482 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.003867 5437 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 01:18:38.009482 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.003869 5437 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 01:18:38.009995 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.003872 5437 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 01:18:38.009995 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.003875 5437 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 01:18:38.009995 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.003877 5437 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 01:18:38.009995 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.003982 5437 flags.go:64] FLAG: --address="0.0.0.0" Nov 03 01:18:38.009995 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004019 5437 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 03 01:18:38.009995 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004027 5437 flags.go:64] FLAG: --anonymous-auth="true" Nov 03 01:18:38.009995 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004032 5437 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 03 01:18:38.009995 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004037 5437 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 03 01:18:38.009995 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004041 5437 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 03 01:18:38.009995 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004046 5437 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 03 01:18:38.009995 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004050 5437 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 03 01:18:38.009995 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004054 5437 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 03 01:18:38.009995 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004057 5437 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 03 01:18:38.009995 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004061 5437 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 03 01:18:38.009995 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004064 5437 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 03 01:18:38.009995 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004067 5437 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 03 01:18:38.009995 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004070 5437 flags.go:64] FLAG: --cgroup-root="" Nov 03 01:18:38.009995 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004073 5437 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 03 01:18:38.009995 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004076 5437 flags.go:64] FLAG: --client-ca-file="" Nov 03 01:18:38.009995 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004079 5437 flags.go:64] FLAG: --cloud-provider="external" Nov 03 01:18:38.009995 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004082 5437 flags.go:64] FLAG: --cluster-dns="[]" Nov 03 01:18:38.009995 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004087 5437 flags.go:64] FLAG: --cluster-domain="" Nov 03 01:18:38.009995 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004090 5437 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 03 01:18:38.010608 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004093 5437 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 03 01:18:38.010608 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004098 5437 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 03 01:18:38.010608 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004102 5437 flags.go:64] FLAG: --container-log-max-files="5" Nov 03 01:18:38.010608 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004106 5437 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 03 01:18:38.010608 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004109 5437 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 03 01:18:38.010608 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004112 5437 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 03 01:18:38.010608 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004117 5437 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 03 01:18:38.010608 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004120 5437 flags.go:64] FLAG: --contention-profiling="false" Nov 03 01:18:38.010608 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004123 5437 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 03 01:18:38.010608 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004127 5437 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 03 01:18:38.010608 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004130 5437 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 03 01:18:38.010608 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004133 5437 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 03 01:18:38.010608 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004137 5437 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 03 01:18:38.010608 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004141 5437 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 03 01:18:38.010608 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004144 5437 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 03 01:18:38.010608 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004147 5437 flags.go:64] FLAG: --enable-load-reader="false" Nov 03 01:18:38.010608 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004150 5437 flags.go:64] FLAG: --enable-server="true" Nov 03 01:18:38.010608 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004153 5437 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 03 01:18:38.010608 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004157 5437 flags.go:64] FLAG: --event-burst="100" Nov 03 01:18:38.010608 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004162 5437 flags.go:64] FLAG: --event-qps="50" Nov 03 01:18:38.010608 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004165 5437 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 03 01:18:38.010608 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004169 5437 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 03 01:18:38.010608 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004172 5437 flags.go:64] FLAG: --eviction-hard="" Nov 03 01:18:38.010608 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004176 5437 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 03 01:18:38.011208 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004179 5437 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 03 01:18:38.011208 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004182 5437 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 03 01:18:38.011208 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004186 5437 flags.go:64] FLAG: --eviction-soft="" Nov 03 01:18:38.011208 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004189 5437 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 03 01:18:38.011208 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004192 5437 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 03 01:18:38.011208 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004195 5437 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 03 01:18:38.011208 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004198 5437 flags.go:64] FLAG: --experimental-mounter-path="" Nov 03 01:18:38.011208 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004202 5437 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 03 01:18:38.011208 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004205 5437 flags.go:64] FLAG: --fail-swap-on="true" Nov 03 01:18:38.011208 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004209 5437 flags.go:64] FLAG: --feature-gates="" Nov 03 01:18:38.011208 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004212 5437 flags.go:64] FLAG: --file-check-frequency="20s" Nov 03 01:18:38.011208 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004216 5437 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 03 01:18:38.011208 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004219 5437 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 03 01:18:38.011208 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004222 5437 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 03 01:18:38.011208 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004226 5437 flags.go:64] FLAG: --healthz-port="10248" Nov 03 01:18:38.011208 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004229 5437 flags.go:64] FLAG: --help="false" Nov 03 01:18:38.011208 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004232 5437 flags.go:64] FLAG: --hostname-override="ip-10-0-136-16.us-west-2.compute.internal" Nov 03 01:18:38.011208 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004236 5437 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 03 01:18:38.011208 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004239 5437 flags.go:64] FLAG: --http-check-frequency="20s" Nov 03 01:18:38.011208 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004242 5437 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 03 01:18:38.011208 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004245 5437 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 03 01:18:38.011208 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004249 5437 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 03 01:18:38.011208 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004252 5437 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 03 01:18:38.011873 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004255 5437 flags.go:64] FLAG: --image-service-endpoint="" Nov 03 01:18:38.011873 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004258 5437 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 03 01:18:38.011873 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004261 5437 flags.go:64] FLAG: --kube-api-burst="100" Nov 03 01:18:38.011873 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004264 5437 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 03 01:18:38.011873 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004267 5437 flags.go:64] FLAG: --kube-api-qps="50" Nov 03 01:18:38.011873 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004270 5437 flags.go:64] FLAG: --kube-reserved="" Nov 03 01:18:38.011873 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004274 5437 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 03 01:18:38.011873 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004276 5437 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 03 01:18:38.011873 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004279 5437 flags.go:64] FLAG: --kubelet-cgroups="" Nov 03 01:18:38.011873 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004282 5437 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 03 01:18:38.011873 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004285 5437 flags.go:64] FLAG: --lock-file="" Nov 03 01:18:38.011873 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004288 5437 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 03 01:18:38.011873 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004291 5437 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 03 01:18:38.011873 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004294 5437 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 03 01:18:38.011873 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004300 5437 flags.go:64] FLAG: --log-json-split-stream="false" Nov 03 01:18:38.011873 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004303 5437 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 03 01:18:38.011873 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004306 5437 flags.go:64] FLAG: --log-text-split-stream="false" Nov 03 01:18:38.011873 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004309 5437 flags.go:64] FLAG: --logging-format="text" Nov 03 01:18:38.011873 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004312 5437 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 03 01:18:38.011873 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004315 5437 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 03 01:18:38.011873 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004318 5437 flags.go:64] FLAG: --manifest-url="" Nov 03 01:18:38.011873 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004321 5437 flags.go:64] FLAG: --manifest-url-header="" Nov 03 01:18:38.011873 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004326 5437 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 03 01:18:38.011873 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004329 5437 flags.go:64] FLAG: --max-open-files="1000000" Nov 03 01:18:38.011873 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004333 5437 flags.go:64] FLAG: --max-pods="110" Nov 03 01:18:38.012475 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004336 5437 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 03 01:18:38.012475 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004339 5437 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 03 01:18:38.012475 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004343 5437 flags.go:64] FLAG: --memory-manager-policy="None" Nov 03 01:18:38.012475 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004346 5437 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 03 01:18:38.012475 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004349 5437 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 03 01:18:38.012475 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004352 5437 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 03 01:18:38.012475 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004356 5437 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 03 01:18:38.012475 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004375 5437 flags.go:64] FLAG: --node-status-max-images="50" Nov 03 01:18:38.012475 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004378 5437 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 03 01:18:38.012475 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004381 5437 flags.go:64] FLAG: --oom-score-adj="-999" Nov 03 01:18:38.012475 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004385 5437 flags.go:64] FLAG: --pod-cidr="" Nov 03 01:18:38.012475 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004388 5437 flags.go:64] FLAG: --pod-manifest-path="" Nov 03 01:18:38.012475 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004390 5437 flags.go:64] FLAG: --pod-max-pids="-1" Nov 03 01:18:38.012475 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004394 5437 flags.go:64] FLAG: --pods-per-core="0" Nov 03 01:18:38.012475 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004397 5437 flags.go:64] FLAG: --port="10250" Nov 03 01:18:38.012475 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004400 5437 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 03 01:18:38.012475 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004404 5437 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-0fe7c81c61095ddcc" Nov 03 01:18:38.012475 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004407 5437 flags.go:64] FLAG: --qos-reserved="" Nov 03 01:18:38.012475 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004410 5437 flags.go:64] FLAG: --read-only-port="10255" Nov 03 01:18:38.012475 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004413 5437 flags.go:64] FLAG: --register-node="true" Nov 03 01:18:38.012475 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004416 5437 flags.go:64] FLAG: --register-with-taints="" Nov 03 01:18:38.012475 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004420 5437 flags.go:64] FLAG: --registry-burst="10" Nov 03 01:18:38.012475 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004423 5437 flags.go:64] FLAG: --registry-qps="5" Nov 03 01:18:38.012475 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004426 5437 flags.go:64] FLAG: --reserved-cpus="" Nov 03 01:18:38.012475 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004429 5437 flags.go:64] FLAG: --reserved-memory="" Nov 03 01:18:38.012475 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004433 5437 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 03 01:18:38.013138 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004436 5437 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 03 01:18:38.013138 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004439 5437 flags.go:64] FLAG: --rotate-certificates="false" Nov 03 01:18:38.013138 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004442 5437 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 03 01:18:38.013138 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004445 5437 flags.go:64] FLAG: --runonce="false" Nov 03 01:18:38.013138 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004448 5437 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 03 01:18:38.013138 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004451 5437 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 03 01:18:38.013138 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004454 5437 flags.go:64] FLAG: --seccomp-default="false" Nov 03 01:18:38.013138 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004457 5437 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 03 01:18:38.013138 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004460 5437 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 03 01:18:38.013138 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004463 5437 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 03 01:18:38.013138 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004466 5437 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 03 01:18:38.013138 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004469 5437 flags.go:64] FLAG: --storage-driver-password="root" Nov 03 01:18:38.013138 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004472 5437 flags.go:64] FLAG: --storage-driver-secure="false" Nov 03 01:18:38.013138 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004475 5437 flags.go:64] FLAG: --storage-driver-table="stats" Nov 03 01:18:38.013138 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004478 5437 flags.go:64] FLAG: --storage-driver-user="root" Nov 03 01:18:38.013138 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004481 5437 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 03 01:18:38.013138 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004484 5437 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 03 01:18:38.013138 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004487 5437 flags.go:64] FLAG: --system-cgroups="" Nov 03 01:18:38.013138 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004490 5437 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 03 01:18:38.013138 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004495 5437 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 03 01:18:38.013138 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004499 5437 flags.go:64] FLAG: --tls-cert-file="" Nov 03 01:18:38.013138 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004502 5437 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 03 01:18:38.013138 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004506 5437 flags.go:64] FLAG: --tls-min-version="" Nov 03 01:18:38.013138 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004509 5437 flags.go:64] FLAG: --tls-private-key-file="" Nov 03 01:18:38.013138 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004512 5437 flags.go:64] FLAG: --topology-manager-policy="none" Nov 03 01:18:38.013760 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004515 5437 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 03 01:18:38.013760 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004518 5437 flags.go:64] FLAG: --topology-manager-scope="container" Nov 03 01:18:38.013760 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004521 5437 flags.go:64] FLAG: --v="2" Nov 03 01:18:38.013760 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004525 5437 flags.go:64] FLAG: --version="false" Nov 03 01:18:38.013760 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004529 5437 flags.go:64] FLAG: --vmodule="" Nov 03 01:18:38.013760 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004533 5437 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 03 01:18:38.013760 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004536 5437 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 03 01:18:38.013760 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.004668 5437 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 01:18:38.013760 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.004672 5437 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 01:18:38.013760 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.004675 5437 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 01:18:38.013760 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.004678 5437 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 01:18:38.013760 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.004681 5437 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 01:18:38.013760 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.004685 5437 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 01:18:38.013760 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.004689 5437 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 01:18:38.013760 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.004692 5437 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 01:18:38.013760 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.004695 5437 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 01:18:38.013760 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.004697 5437 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 01:18:38.013760 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.004700 5437 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 01:18:38.013760 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.004702 5437 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 01:18:38.013760 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.004705 5437 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 01:18:38.013760 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.004708 5437 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 01:18:38.014253 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.004712 5437 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 01:18:38.014253 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.004715 5437 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 01:18:38.014253 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.004717 5437 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 01:18:38.014253 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.004720 5437 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 01:18:38.014253 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.004722 5437 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 01:18:38.014253 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.004725 5437 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 01:18:38.014253 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.004728 5437 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 01:18:38.014253 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.004731 5437 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 01:18:38.014253 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.004734 5437 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 01:18:38.014253 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.004736 5437 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 01:18:38.014253 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.004740 5437 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 01:18:38.014253 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.004743 5437 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 01:18:38.014253 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.004746 5437 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 01:18:38.014253 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.004748 5437 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 01:18:38.014253 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.004751 5437 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 01:18:38.014253 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.004753 5437 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 01:18:38.014253 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.004756 5437 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 01:18:38.014253 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.004759 5437 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 01:18:38.014253 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.004761 5437 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 01:18:38.014760 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.004764 5437 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 01:18:38.014760 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.004766 5437 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 01:18:38.014760 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.004769 5437 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 01:18:38.014760 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.004771 5437 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 01:18:38.014760 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.004774 5437 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 01:18:38.014760 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.004777 5437 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 01:18:38.014760 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.004779 5437 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 01:18:38.014760 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.004782 5437 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 01:18:38.014760 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.004784 5437 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 01:18:38.014760 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.004787 5437 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 01:18:38.014760 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.004790 5437 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 01:18:38.014760 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.004793 5437 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 01:18:38.014760 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.004796 5437 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 01:18:38.014760 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.004800 5437 feature_gate.go:386] unrecognized feature gate: Example Nov 03 01:18:38.014760 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.004803 5437 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 01:18:38.014760 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.004805 5437 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 01:18:38.014760 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.004808 5437 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 01:18:38.014760 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.004811 5437 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 01:18:38.014760 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.004813 5437 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 01:18:38.014760 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.004816 5437 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 01:18:38.015312 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.004819 5437 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 01:18:38.015312 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.004822 5437 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 01:18:38.015312 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.004824 5437 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 01:18:38.015312 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.004828 5437 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 01:18:38.015312 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.004831 5437 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 01:18:38.015312 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.004834 5437 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 01:18:38.015312 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.004837 5437 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 01:18:38.015312 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.004839 5437 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 01:18:38.015312 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.004842 5437 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 01:18:38.015312 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.004844 5437 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 01:18:38.015312 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.004847 5437 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 01:18:38.015312 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.004850 5437 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 01:18:38.015312 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.004852 5437 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 01:18:38.015312 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.004855 5437 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 01:18:38.015312 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.004858 5437 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 01:18:38.015312 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.004860 5437 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 01:18:38.015312 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.004863 5437 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 01:18:38.015312 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.004865 5437 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 01:18:38.015312 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.004868 5437 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 01:18:38.015312 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.004870 5437 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 01:18:38.015828 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.004873 5437 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 01:18:38.015828 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.004875 5437 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 01:18:38.015828 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.004878 5437 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 01:18:38.015828 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.004881 5437 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 01:18:38.015828 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.004883 5437 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 01:18:38.015828 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.004889 5437 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 01:18:38.015828 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.004893 5437 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 01:18:38.015828 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.004896 5437 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 01:18:38.015828 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.004899 5437 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 01:18:38.015828 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.004902 5437 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 01:18:38.015828 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.004905 5437 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 01:18:38.015828 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.004907 5437 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 01:18:38.015828 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.004910 5437 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 01:18:38.015828 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.004912 5437 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 01:18:38.015828 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.004916 5437 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 01:18:38.015828 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.004920 5437 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 01:18:38.015828 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.004922 5437 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 01:18:38.015828 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.004925 5437 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 01:18:38.015828 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.004927 5437 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 01:18:38.016291 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.004930 5437 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 01:18:38.016291 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.004933 5437 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 01:18:38.016291 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.004935 5437 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 01:18:38.016291 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.004938 5437 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 01:18:38.016291 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.004940 5437 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 01:18:38.016291 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.004943 5437 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 01:18:38.016291 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.004945 5437 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 01:18:38.016291 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.004948 5437 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 01:18:38.016291 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.004951 5437 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 01:18:38.016291 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.004953 5437 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 01:18:38.016291 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.004956 5437 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 01:18:38.016291 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.004959 5437 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 01:18:38.016291 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.004962 5437 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 01:18:38.016291 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.004964 5437 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 01:18:38.016291 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.004990 5437 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 01:18:38.016291 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.005022 5437 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 01:18:38.016747 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.005027 5437 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 01:18:38.016747 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.005030 5437 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 01:18:38.016747 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.005032 5437 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 01:18:38.016747 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.005036 5437 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 01:18:38.016747 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.005039 5437 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 01:18:38.016747 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.005042 5437 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 01:18:38.016747 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.005045 5437 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 01:18:38.016747 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.005047 5437 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 01:18:38.016747 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.005050 5437 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 01:18:38.016747 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.005053 5437 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 01:18:38.016747 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.005056 5437 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 01:18:38.016747 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.005058 5437 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 01:18:38.016747 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.005063 5437 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 01:18:38.016747 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.005066 5437 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 01:18:38.016747 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.005068 5437 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 01:18:38.016747 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.005071 5437 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 01:18:38.016747 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.005073 5437 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 01:18:38.016747 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.005076 5437 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 01:18:38.016747 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.005079 5437 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 01:18:38.017196 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.005081 5437 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 01:18:38.017196 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.005084 5437 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 01:18:38.017196 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.005086 5437 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 01:18:38.017196 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.005089 5437 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 01:18:38.017196 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.005091 5437 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 01:18:38.017196 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.005094 5437 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 01:18:38.017196 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.005097 5437 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 01:18:38.017196 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.005099 5437 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 01:18:38.017196 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.005102 5437 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 01:18:38.017196 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.005104 5437 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 01:18:38.017196 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.005107 5437 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 01:18:38.017196 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.005109 5437 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 01:18:38.017196 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.005112 5437 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 01:18:38.017196 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.005114 5437 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 01:18:38.017196 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.005117 5437 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 01:18:38.017196 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.005119 5437 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 01:18:38.017196 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.005123 5437 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 01:18:38.017196 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.005126 5437 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 01:18:38.017196 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.005129 5437 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 01:18:38.017196 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.005132 5437 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 01:18:38.017720 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.005135 5437 feature_gate.go:386] unrecognized feature gate: Example Nov 03 01:18:38.017720 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.005138 5437 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 01:18:38.017720 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.005140 5437 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 01:18:38.017720 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.005143 5437 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 01:18:38.017720 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.005146 5437 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 01:18:38.017720 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.005149 5437 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 01:18:38.017720 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.005152 5437 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 01:18:38.017720 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.005155 5437 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 01:18:38.017720 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.005158 5437 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 01:18:38.017720 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.005161 5437 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 01:18:38.017720 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.005164 5437 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 01:18:38.017720 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.005167 5437 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 01:18:38.017720 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.005169 5437 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 01:18:38.017720 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.005172 5437 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 01:18:38.017720 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.005175 5437 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 01:18:38.017720 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.005177 5437 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 01:18:38.017720 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.005180 5437 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 01:18:38.017720 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.005183 5437 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 01:18:38.017720 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.005185 5437 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 01:18:38.017720 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.005188 5437 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 01:18:38.018488 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.005191 5437 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 01:18:38.018488 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.005194 5437 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 01:18:38.018488 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.005196 5437 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 01:18:38.018488 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.005199 5437 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 01:18:38.018488 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.005201 5437 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 01:18:38.018488 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.005204 5437 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 01:18:38.018488 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.005207 5437 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 01:18:38.018488 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.005209 5437 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 01:18:38.018488 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.005213 5437 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 01:18:38.018488 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.005216 5437 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 01:18:38.018488 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.005218 5437 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 01:18:38.018488 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.005221 5437 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 01:18:38.018488 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.005225 5437 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 01:18:38.018488 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.005229 5437 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 01:18:38.018488 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.005232 5437 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 01:18:38.018488 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.005235 5437 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 01:18:38.018488 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.005238 5437 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 01:18:38.018488 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.005242 5437 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 01:18:38.018488 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.005245 5437 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 01:18:38.019064 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.005248 5437 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 01:18:38.019064 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.005251 5437 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 01:18:38.019064 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.005254 5437 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 01:18:38.019064 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.005256 5437 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 01:18:38.019064 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.005259 5437 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 01:18:38.019064 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.005261 5437 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 01:18:38.019064 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.005264 5437 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 01:18:38.019064 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.005267 5437 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 01:18:38.019064 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.005269 5437 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 01:18:38.019064 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.005272 5437 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 01:18:38.019064 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.005274 5437 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 01:18:38.019064 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.005277 5437 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 01:18:38.019064 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.005280 5437 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 01:18:38.019064 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.005282 5437 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 01:18:38.019064 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.005285 5437 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 01:18:38.019064 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.005287 5437 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 01:18:38.019064 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.005290 5437 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 01:18:38.019064 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.005293 5437 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 01:18:38.019064 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.005296 5437 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 01:18:38.019064 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.005298 5437 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 01:18:38.019570 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.005301 5437 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 01:18:38.019570 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.005305 5437 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 01:18:38.019570 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.005307 5437 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 01:18:38.019570 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.005310 5437 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 01:18:38.019570 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.005313 5437 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 01:18:38.019570 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.005315 5437 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 01:18:38.019570 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.005320 5437 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 01:18:38.019570 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.010218 5437 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 03 01:18:38.019570 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.010237 5437 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 03 01:18:38.019570 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.010261 5437 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 03 01:18:38.019570 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.010266 5437 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 03 01:18:38.019570 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010326 5437 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 01:18:38.019570 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010332 5437 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 01:18:38.019570 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010335 5437 feature_gate.go:386] unrecognized feature gate: Example Nov 03 01:18:38.019570 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010338 5437 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 01:18:38.019570 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010341 5437 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 01:18:38.019570 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010344 5437 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 01:18:38.019570 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010347 5437 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 01:18:38.019570 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010350 5437 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 01:18:38.019570 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010353 5437 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 01:18:38.020052 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010356 5437 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 01:18:38.020052 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010359 5437 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 01:18:38.020052 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010402 5437 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 01:18:38.020052 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010408 5437 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 01:18:38.020052 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010412 5437 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 01:18:38.020052 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010415 5437 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 01:18:38.020052 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010418 5437 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 01:18:38.020052 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010421 5437 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 01:18:38.020052 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010424 5437 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 01:18:38.020052 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010429 5437 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 01:18:38.020052 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010432 5437 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 01:18:38.020052 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010436 5437 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 01:18:38.020052 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010439 5437 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 01:18:38.020052 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010442 5437 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 01:18:38.020052 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010446 5437 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 01:18:38.020052 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010448 5437 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 01:18:38.020052 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010451 5437 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 01:18:38.020052 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010454 5437 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 01:18:38.020052 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010457 5437 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 01:18:38.020052 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010460 5437 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 01:18:38.020555 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010463 5437 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 01:18:38.020555 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010465 5437 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 01:18:38.020555 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010468 5437 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 01:18:38.020555 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010471 5437 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 01:18:38.020555 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010474 5437 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 01:18:38.020555 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010476 5437 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 01:18:38.020555 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010479 5437 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 01:18:38.020555 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010482 5437 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 01:18:38.020555 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010484 5437 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 01:18:38.020555 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010487 5437 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 01:18:38.020555 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010489 5437 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 01:18:38.020555 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010492 5437 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 01:18:38.020555 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010495 5437 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 01:18:38.020555 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010498 5437 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 01:18:38.020555 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010501 5437 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 01:18:38.020555 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010505 5437 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 01:18:38.020555 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010508 5437 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 01:18:38.020555 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010511 5437 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 01:18:38.020555 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010514 5437 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 01:18:38.022574 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010517 5437 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 01:18:38.022574 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010520 5437 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 01:18:38.022574 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010523 5437 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 01:18:38.022574 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010526 5437 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 01:18:38.022574 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010529 5437 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 01:18:38.022574 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010533 5437 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 01:18:38.022574 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010536 5437 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 01:18:38.022574 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010539 5437 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 01:18:38.022574 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010543 5437 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 01:18:38.022574 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010546 5437 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 01:18:38.022574 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010549 5437 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 01:18:38.022574 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010552 5437 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 01:18:38.022574 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010554 5437 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 01:18:38.022574 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010557 5437 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 01:18:38.022574 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010560 5437 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 01:18:38.022574 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010562 5437 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 01:18:38.022574 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010565 5437 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 01:18:38.022574 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010568 5437 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 01:18:38.022574 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010571 5437 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 01:18:38.023036 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010574 5437 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 01:18:38.023036 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010576 5437 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 01:18:38.023036 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010579 5437 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 01:18:38.023036 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010581 5437 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 01:18:38.023036 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010584 5437 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 01:18:38.023036 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010587 5437 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 01:18:38.023036 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010590 5437 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 01:18:38.023036 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010593 5437 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 01:18:38.023036 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010595 5437 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 01:18:38.023036 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010598 5437 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 01:18:38.023036 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010601 5437 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 01:18:38.023036 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010604 5437 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 01:18:38.023036 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010607 5437 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 01:18:38.023036 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010610 5437 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 01:18:38.023036 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010613 5437 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 01:18:38.023036 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010616 5437 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 01:18:38.023036 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010619 5437 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 01:18:38.023036 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010622 5437 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 01:18:38.023036 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010625 5437 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 01:18:38.023529 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010627 5437 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 01:18:38.023529 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010630 5437 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 01:18:38.023529 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010632 5437 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 01:18:38.023529 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010635 5437 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 01:18:38.023529 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010638 5437 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 01:18:38.023529 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010641 5437 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 01:18:38.023529 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010643 5437 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 01:18:38.023529 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010646 5437 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 01:18:38.023529 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010649 5437 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 01:18:38.023529 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010652 5437 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 01:18:38.023529 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010654 5437 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 01:18:38.023529 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010657 5437 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 01:18:38.023529 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010660 5437 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 01:18:38.023529 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010662 5437 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 01:18:38.023529 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010665 5437 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 01:18:38.023529 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010667 5437 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 01:18:38.023529 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010670 5437 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 01:18:38.023529 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010672 5437 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 01:18:38.023529 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010675 5437 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 01:18:38.023529 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010679 5437 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 01:18:38.024013 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.010709 5437 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 01:18:38.024013 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010852 5437 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 01:18:38.024013 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010858 5437 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 01:18:38.024013 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010861 5437 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 01:18:38.024013 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010865 5437 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 01:18:38.024013 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010868 5437 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 01:18:38.024013 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010872 5437 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 01:18:38.024013 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010874 5437 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 01:18:38.024013 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010877 5437 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 01:18:38.024013 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010880 5437 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 01:18:38.024013 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010882 5437 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 01:18:38.024013 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010885 5437 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 01:18:38.024013 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010888 5437 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 01:18:38.024013 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010890 5437 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 01:18:38.024013 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010893 5437 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 01:18:38.024013 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010896 5437 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 01:18:38.024416 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010898 5437 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 01:18:38.024416 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010901 5437 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 01:18:38.024416 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010904 5437 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 01:18:38.024416 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010907 5437 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 01:18:38.024416 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010909 5437 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 01:18:38.024416 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010912 5437 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 01:18:38.024416 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010914 5437 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 01:18:38.024416 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010917 5437 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 01:18:38.024416 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010919 5437 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 01:18:38.024416 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010922 5437 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 01:18:38.024416 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010924 5437 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 01:18:38.024416 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010927 5437 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 01:18:38.024416 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010929 5437 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 01:18:38.024416 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010932 5437 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 01:18:38.024416 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010935 5437 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 01:18:38.024416 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010938 5437 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 01:18:38.024416 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010940 5437 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 01:18:38.024416 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010943 5437 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 01:18:38.024416 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010945 5437 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 01:18:38.024416 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010948 5437 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 01:18:38.024891 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010950 5437 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 01:18:38.024891 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010953 5437 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 01:18:38.024891 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010956 5437 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 01:18:38.024891 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010959 5437 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 01:18:38.024891 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010962 5437 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 01:18:38.024891 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010965 5437 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 01:18:38.024891 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010967 5437 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 01:18:38.024891 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010970 5437 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 01:18:38.024891 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010972 5437 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 01:18:38.024891 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010975 5437 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 01:18:38.024891 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010977 5437 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 01:18:38.024891 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010980 5437 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 01:18:38.024891 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010982 5437 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 01:18:38.024891 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010985 5437 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 01:18:38.024891 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010988 5437 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 01:18:38.024891 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010990 5437 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 01:18:38.024891 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010993 5437 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 01:18:38.024891 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010995 5437 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 01:18:38.024891 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.010998 5437 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 01:18:38.025346 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011000 5437 feature_gate.go:386] unrecognized feature gate: Example Nov 03 01:18:38.025346 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011003 5437 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 01:18:38.025346 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011006 5437 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 01:18:38.025346 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011009 5437 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 01:18:38.025346 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011011 5437 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 01:18:38.025346 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011014 5437 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 01:18:38.025346 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011016 5437 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 01:18:38.025346 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011019 5437 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 01:18:38.025346 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011022 5437 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 01:18:38.025346 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011025 5437 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 01:18:38.025346 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011027 5437 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 01:18:38.025346 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011030 5437 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 01:18:38.025346 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011032 5437 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 01:18:38.025346 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011035 5437 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 01:18:38.025346 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011038 5437 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 01:18:38.025346 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011040 5437 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 01:18:38.025346 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011043 5437 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 01:18:38.025346 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011050 5437 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 01:18:38.025346 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011053 5437 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 01:18:38.025828 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011057 5437 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 01:18:38.025828 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011061 5437 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 01:18:38.025828 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011064 5437 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 01:18:38.025828 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011066 5437 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 01:18:38.025828 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011069 5437 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 01:18:38.025828 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011071 5437 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 01:18:38.025828 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011074 5437 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 01:18:38.025828 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011079 5437 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 01:18:38.025828 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011082 5437 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 01:18:38.025828 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011085 5437 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 01:18:38.025828 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011087 5437 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 01:18:38.025828 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011090 5437 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 01:18:38.025828 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011093 5437 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 01:18:38.025828 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011096 5437 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 01:18:38.025828 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011098 5437 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 01:18:38.025828 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011101 5437 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 01:18:38.025828 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011103 5437 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 01:18:38.025828 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011106 5437 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 01:18:38.025828 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011109 5437 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 01:18:38.026291 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011112 5437 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 01:18:38.026291 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011114 5437 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 01:18:38.026291 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011117 5437 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 01:18:38.026291 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011120 5437 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 01:18:38.026291 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011123 5437 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 01:18:38.026291 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011125 5437 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 01:18:38.026291 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011128 5437 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 01:18:38.026291 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011130 5437 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 01:18:38.026291 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011133 5437 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 01:18:38.026291 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011136 5437 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 01:18:38.026291 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011138 5437 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 01:18:38.026291 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011141 5437 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 01:18:38.026291 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011144 5437 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 01:18:38.026291 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011147 5437 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 01:18:38.026291 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.011169 5437 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 01:18:38.026291 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011201 5437 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 01:18:38.026702 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011205 5437 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 01:18:38.026702 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011208 5437 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 01:18:38.026702 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011210 5437 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 01:18:38.026702 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011213 5437 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 01:18:38.026702 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011216 5437 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 01:18:38.026702 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011219 5437 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 01:18:38.026702 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011223 5437 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 01:18:38.026702 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011226 5437 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 01:18:38.026702 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011228 5437 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 01:18:38.026702 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011231 5437 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 01:18:38.026702 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011234 5437 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 01:18:38.026702 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011236 5437 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 01:18:38.026702 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011239 5437 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 01:18:38.026702 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011242 5437 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 01:18:38.026702 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011244 5437 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 01:18:38.026702 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011247 5437 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 01:18:38.026702 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011250 5437 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 01:18:38.026702 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011252 5437 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 01:18:38.026702 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011255 5437 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 01:18:38.027236 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011257 5437 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 01:18:38.027236 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011260 5437 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 01:18:38.027236 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011263 5437 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 01:18:38.027236 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011265 5437 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 01:18:38.027236 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011268 5437 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 01:18:38.027236 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011270 5437 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 01:18:38.027236 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011273 5437 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 01:18:38.027236 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011275 5437 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 01:18:38.027236 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011278 5437 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 01:18:38.027236 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011280 5437 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 01:18:38.027236 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011284 5437 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 01:18:38.027236 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011286 5437 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 01:18:38.027236 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011289 5437 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 01:18:38.027236 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011292 5437 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 01:18:38.027236 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011294 5437 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 01:18:38.027236 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011297 5437 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 01:18:38.027236 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011300 5437 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 01:18:38.027236 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011303 5437 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 01:18:38.027236 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011306 5437 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 01:18:38.027736 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011308 5437 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 01:18:38.027736 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011311 5437 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 01:18:38.027736 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011314 5437 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 01:18:38.027736 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011316 5437 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 01:18:38.027736 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011319 5437 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 01:18:38.027736 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011321 5437 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 01:18:38.027736 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011324 5437 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 01:18:38.027736 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011326 5437 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 01:18:38.027736 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011329 5437 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 01:18:38.027736 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011331 5437 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 01:18:38.027736 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011334 5437 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 01:18:38.027736 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011336 5437 feature_gate.go:386] unrecognized feature gate: Example Nov 03 01:18:38.027736 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011339 5437 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 01:18:38.027736 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011342 5437 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 01:18:38.027736 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011344 5437 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 01:18:38.027736 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011347 5437 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 01:18:38.027736 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011350 5437 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 01:18:38.027736 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011352 5437 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 01:18:38.027736 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011355 5437 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 01:18:38.027736 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011357 5437 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 01:18:38.028213 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011375 5437 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 01:18:38.028213 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011380 5437 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 01:18:38.028213 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011384 5437 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 01:18:38.028213 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011387 5437 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 01:18:38.028213 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011390 5437 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 01:18:38.028213 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011393 5437 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 01:18:38.028213 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011396 5437 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 01:18:38.028213 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011399 5437 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 01:18:38.028213 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011401 5437 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 01:18:38.028213 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011404 5437 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 01:18:38.028213 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011407 5437 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 01:18:38.028213 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011411 5437 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 01:18:38.028213 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011414 5437 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 01:18:38.028213 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011417 5437 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 01:18:38.028213 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011419 5437 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 01:18:38.028213 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011429 5437 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 01:18:38.028213 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011432 5437 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 01:18:38.028213 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011435 5437 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 01:18:38.028213 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011437 5437 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 01:18:38.028726 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011440 5437 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 01:18:38.028726 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011442 5437 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 01:18:38.028726 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011445 5437 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 01:18:38.028726 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011447 5437 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 01:18:38.028726 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011451 5437 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 01:18:38.028726 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011454 5437 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 01:18:38.028726 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011457 5437 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 01:18:38.028726 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011459 5437 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 01:18:38.028726 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011462 5437 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 01:18:38.028726 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011465 5437 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 01:18:38.028726 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011467 5437 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 01:18:38.028726 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011470 5437 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 01:18:38.028726 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011473 5437 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 01:18:38.028726 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011475 5437 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 01:18:38.028726 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011478 5437 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 01:18:38.028726 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011481 5437 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 01:18:38.028726 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011483 5437 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 01:18:38.028726 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011486 5437 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 01:18:38.028726 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011489 5437 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 01:18:38.028726 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011491 5437 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 01:18:38.029212 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011494 5437 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 01:18:38.029212 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011497 5437 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 01:18:38.029212 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011499 5437 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 01:18:38.029212 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011502 5437 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 01:18:38.029212 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011506 5437 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 01:18:38.029212 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011509 5437 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 01:18:38.029212 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011512 5437 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 01:18:38.029212 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:18:38.011515 5437 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 01:18:38.029212 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.011577 5437 server.go:957] "Client rotation is on, will bootstrap in background" Nov 03 01:18:38.029212 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.012556 5437 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 03 01:18:38.029212 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.012615 5437 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 03 01:18:38.029212 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.012793 5437 server.go:1014] "Starting client certificate rotation" Nov 03 01:18:38.029212 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.012946 5437 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 01:18:38.029212 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.013006 5437 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 01:12:25 +0000 UTC" deadline="2028-06-12 17:32:40.905962936 +0000 UTC" Nov 03 01:18:38.029212 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.013060 5437 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14104h14m2.892904923s" Nov 03 01:18:38.029212 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.013271 5437 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 01:18:38.029612 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.014309 5437 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 01:18:38.029612 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.015756 5437 log.go:25] "Validated CRI v1 runtime API" Nov 03 01:18:38.029612 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.017677 5437 log.go:25] "Validated CRI v1 image API" Nov 03 01:18:38.029612 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.018527 5437 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 03 01:18:38.029612 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.020665 5437 fs.go:135] Filesystem UUIDs: map[3cb574a7-0402-4d32-a20a-db0a7770476a:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 8a1ba601-aa3f-4674-8909-74086aba29ec:/dev/nvme0n1p4] Nov 03 01:18:38.029807 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.020685 5437 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0e9e7edee56b546df74773696b5a5d39e39e5a667ed6d072684745e5f19334ff/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0e9e7edee56b546df74773696b5a5d39e39e5a667ed6d072684745e5f19334ff/userdata/shm major:0 minor:251 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1e4ab311b886828acac64bcbb2ec9807a19f1a8f64a2faaf1ba578edef09bfd1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1e4ab311b886828acac64bcbb2ec9807a19f1a8f64a2faaf1ba578edef09bfd1/userdata/shm major:0 minor:385 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/24a6b7dfc849cfef3310febb8cba625fb2635841a98b3907ddfe9a3951c334ba/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/24a6b7dfc849cfef3310febb8cba625fb2635841a98b3907ddfe9a3951c334ba/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/24bd13d85cff8f0cf46be8ea2d05895b41b5b5beed80366bec3a1469068ed1ab/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/24bd13d85cff8f0cf46be8ea2d05895b41b5b5beed80366bec3a1469068ed1ab/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/31db35db8133fd4c1b3f0909bcef183472dca46c17fb140c4240f9c733b82412/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/31db35db8133fd4c1b3f0909bcef183472dca46c17fb140c4240f9c733b82412/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3718cadbf973e67ccd8645ba1efff38c7704ba18c1a9ca4d2d3c853a7544a95d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3718cadbf973e67ccd8645ba1efff38c7704ba18c1a9ca4d2d3c853a7544a95d/userdata/shm major:0 minor:386 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3a98ca3c6966d3e44ce6dcd1ee13bd9cdf5b46bbeb0c34fe97865c015118ff10/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3a98ca3c6966d3e44ce6dcd1ee13bd9cdf5b46bbeb0c34fe97865c015118ff10/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/406839857f2f8eaa16b794d2e388372be8131e40e4df221e5bea5891f1d3cd05/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/406839857f2f8eaa16b794d2e388372be8131e40e4df221e5bea5891f1d3cd05/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4d1338fb1357347638c76068111fb2982b81469361736cb026758c7679f468a2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4d1338fb1357347638c76068111fb2982b81469361736cb026758c7679f468a2/userdata/shm major:0 minor:304 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/640208824d01bacb81bae87037315951d0d6301e18043589626cbb13a7a78d2b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/640208824d01bacb81bae87037315951d0d6301e18043589626cbb13a7a78d2b/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7c7251f3fe76aa5de4ec3a3488313d38429bb12e51be59710fb3d62113862144/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7c7251f3fe76aa5de4ec3a3488313d38429bb12e51be59710fb3d62113862144/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7e4535033e2187d3a887d160f43211e0b3f012c6b23c0b04d27c966b04e03c50/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7e4535033e2187d3a887d160f43211e0b3f012c6b23c0b04d27c966b04e03c50/userdata/shm major:0 minor:355 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8faf3e8cf655ef04445ca37b09e02a7a6303ffbe2ea86a0fd98a8e918838d88d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8faf3e8cf655ef04445ca37b09e02a7a6303ffbe2ea86a0fd98a8e918838d88d/userdata/shm major:0 minor:203 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/97a126238281ad8365de918d31b44613980842c7d124f305133855532128b7c3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/97a126238281ad8365de918d31b44613980842c7d124f305133855532128b7c3/userdata/shm major:0 minor:326 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a0a9ec2bdcdd88a98fc92b6fa9724aac0472c842079f5b018614d7d35bef61a2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a0a9ec2bdcdd88a98fc92b6fa9724aac0472c842079f5b018614d7d35bef61a2/userdata/shm major:0 minor:250 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a57cfab1072e0e4b06e4b4217747174d1f2639b037b9403811af934a57dc23e5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a57cfab1072e0e4b06e4b4217747174d1f2639b037b9403811af934a57dc23e5/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a8e0819cc9cee4743891c1bd3a02cdedd551a431d84689742656f35cef6c34de/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a8e0819cc9cee4743891c1bd3a02cdedd551a431d84689742656f35cef6c34de/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b47a157f9083e86acb094f9f6f448c186b115df3f282b88bae7bb034ef72ee4e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b47a157f9083e86acb094f9f6f448c186b115df3f282b88bae7bb034ef72ee4e/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/beaf53b14654cf966260cc8f6b640829a5f2cbd5414b1aa7902e5fb960ffd8fb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/beaf53b14654cf966260cc8f6b640829a5f2cbd5414b1aa7902e5fb960ffd8fb/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ce3ef08fafa3dfc7c74905d661a123ef822c6af06a08d3f565ed2340bed81e4a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ce3ef08fafa3dfc7c74905d661a123ef822c6af06a08d3f565ed2340bed81e4a/userdata/shm major:0 minor:383 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ec11b93955fdbcf32dfeefb15d657d495d24cb563f8d5b07067303a2cc67c40a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ec11b93955fdbcf32dfeefb15d657d495d24cb563f8d5b07067303a2cc67c40a/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f1bbf393f14b3c0361a059eb428e4657c6a8c03af1b7fc3416c78ecb0a1423f1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f1bbf393f14b3c0361a059eb428e4657c6a8c03af1b7fc3416c78ecb0a1423f1/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0e35c4cd-ef18-40c1-8568-f77773e378b2/volumes/kubernetes.io~projected/kube-api-access-shwrh:{mountpoint:/var/lib/kubelet/pods/0e35c4cd-ef18-40c1-8568-f77773e378b2/volumes/kubernetes.io~projected/kube-api-access-shwrh major:0 minor:241 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0e35c4cd-ef18-40c1-8568-f77773e378b2/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/0e35c4cd-ef18-40c1-8568-f77773e378b2/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config major:0 minor:240 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0e35c4cd-ef18-40c1-8568-f77773e378b2/volumes/kubernetes.io~secret/openshift-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/0e35c4cd-ef18-40c1-8568-f77773e378b2/volumes/kubernetes.io~secret/openshift-state-metrics-tls major:0 minor:236 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1fdb6f2d-9138-4f57-af12-c3909dff6e5f/volumes/kubernetes.io~projected/kube-api-access-8flcc:{mountpoint:/var/lib/kubelet/pods/1fdb6f2d-9138-4f57-af12-c3909dff6e5f/volumes/kubernetes.io~projected/kube-api-access-8flcc major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1fdb6f2d-9138-4f57-af12-c3909dff6e5f/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/1fdb6f2d-9138-4f57-af12-c3909dff6e5f/volumes/kubernetes.io~secret/metrics-tls major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1febd8fa-5641-449b-b1fd-c7d02603fd1c/volumes/kubernetes.io~projected/kube-api-access-bdm2b:{mountpoint:/var/lib/kubelet/pods/1febd8fa-5641-449b-b1fd-c7d02603fd1c/volumes/kubernetes.io~projected/kube-api-access-bdm2b major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/20152fc3-0b5f-4c58-94fb-e02094377c74/volumes/kubernetes.io~projected/kube-api-access-m2r8s:{mountpoint:/var/lib/kubelet/pods/20152fc3-0b5f-4c58-94fb-e02094377c74/volumes/kubernetes.io~projected/kube-api-access-m2r8s major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2320cce9-0fdb-470a-ae01-e88118ba0419/volumes/kubernetes.io~projected/kube-api-access-b2cz9:{mountpoint:/var/lib/kubelet/pods/2320cce9-0fdb-470a-ae01-e88118ba0419/volumes/kubernetes.io~projected/kube-api-access-b2cz9 major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2320cce9-0fdb-470a-ae01-e88118ba0419/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/2320cce9-0fdb-470a-ae01-e88118ba0419/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/30eede54-1970-4f52-b219-70d3c469ddd4/volumes/kubernetes.io~projected/kube-api-access-sq5fx:{mountpoint:/var/lib/kubelet/pods/30eede54-1970-4f52-b219-70d3c469ddd4/volumes/kubernetes.io~projected/kube-api-access-sq5fx major:0 minor:249 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/30eede54-1970-4f52-b219-70d3c469ddd4/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/30eede54-1970-4f52-b219-70d3c469ddd4/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:247 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/30eede54-1970-4f52-b219-70d3c469ddd4/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/30eede54-1970-4f52-b219-70d3c469ddd4/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:246 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/386397a9-5c8d-4221-894b-279e0b396afe/volumes/kubernetes.io~projected/kube-api-access-wk6dw:{mountpoint:/var/lib/kubelet/pods/386397a9-5c8d-4221-894b-279e0b396afe/volumes/kubernetes.io~projected/kube-api-access-wk6dw major:0 minor:354 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/386397a9-5c8d-4221-894b-279e0b396afe/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/386397a9-5c8d-4221-894b-279e0b396afe/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:348 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/386397a9-5c8d-4221-894b-279e0b396afe/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/386397a9-5c8d-4221-894b-279e0b396afe/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:353 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3b2bf2f6-28aa-4480-9f4a-f4e7f659e966/volumes/kubernetes.io~projected/kube-api-access-vgd8w:{mountpoint:/var/lib/kubelet/pods/3b2bf2f6-28aa-4480-9f4a-f4e7f659e966/volumes/kubernetes.io~projected/kube-api-access-vgd8w major:0 minor:382 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4c25011b-c85c-4736-9f72-842a96949072/volumes/kubernetes.io~projected/kube-api-access-vzr84:{mountpoint:/var/lib/kubelet/pods/4c25011b-c85c-4736-9f72-842a96949072/volumes/kubernetes.io~projected/kube-api-access-vzr84 major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6179266f-07ff-48cd-bf03-3f1e961529d7/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/6179266f-07ff-48cd-bf03-3f1e961529d7/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:381 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6ecc2615-b458-41f8-839f-e22c53e7ad0e/volumes/kubernetes.io~projected/kube-api-access-b56kh:{mountpoint:/var/lib/kubelet/pods/6ecc2615-b458-41f8-839f-e22c53e7ad0e/volumes/kubernetes.io~projected/kube-api-access-b56kh major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6ecc2615-b458-41f8-839f-e22c53e7ad0e/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/6ecc2615-b458-41f8-839f-e22c53e7ad0e/volumes/kubernetes.io~secret/metrics-certs major:0 minor:376 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/766a044f-fc66-4998-8006-343c0329860d/volumes/kubernetes.io~projected/kube-api-access-b7ckb:{mountpoint:/var/lib/kubelet/pods/766a044f-fc66-4998-8006-343c0329860d/volumes/kubernetes.io~projected/kube-api-access-b7ckb major:0 minor:248 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/766a044f-fc66-4998-8006-343c0329860d/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/766a044f-fc66-4998-8006-343c0329860d/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config major:0 minor:245 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/766a044f-fc66-4998-8006-343c0329860d/volumes/kubernetes.io~secret/kube-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/766a044f-fc66-4998-8006-343c0329860d/volumes/kubernetes.io~secret/kube-state-metrics-tls major:0 minor:244 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8f478179-5984-48dd-b872-8f552954ecea/volumes/kubernetes.io~projected/kube-api-access-v8l47:{mountpoint:/var/lib/kubelet/pods/8f478179-5984-48dd-b872-8f552954ecea/volumes/kubernetes.io~projected/kube-api-access-v8l47 major:0 minor:303 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8f478179-5984-48dd-b872-8f552954ecea/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/8f478179-5984-48dd-b872-8f552954ecea/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:299 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8f478179-5984-48dd-b872-8f552954ecea/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/8f478179-5984-48dd-b872-8f552954ecea/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy major:0 minor:294 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8f478179-5984-48dd-b872-8f552954ecea/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics:{mountpoint:/var/lib/kubelet/pods/8f478179-5984-48dd-b872-8f552954ecea/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics major:0 minor:302 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8f478179-5984-48dd-b872-8f552954ecea/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules:{mountpoint:/var/lib/kubelet/pods/8f478179-5984-48dd-b872-8f552954ecea/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules major:0 minor:301 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8f478179-5984-48dd-b872-8f552954ecea/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/8f478179-5984-48dd-b872-8f552954ecea/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web major:0 minor:298 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8f478179-5984-48dd-b872-8f552954ecea/volumes/kubernetes.io~secret/secret-thanos-querier-tls:{mountpoint:/var/lib/kubelet/pods/8f478179-5984-48dd-b872-8f552954ecea/volumes/kubernetes.io~secret/secret-thanos-querier-tls major:0 minor:300 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/97cb83aa-8922-4221-8407-fbbb6c39f5c7/volumes/kubernetes.io~projected/kube-api-access-4v9cn:{mountpoint:/var/lib/kubelet/pods/97cb83aa-8922-4221-8407-fbbb6c39f5c7/volumes/kubernetes.io~projected/kube-api-access-4v9cn major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/97cb83aa-8922-4221-8407-fbbb6c39f5c7/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/97cb83aa-8922-4221-8407-fbbb6c39f5c7/volumes/kubernetes.io~secret/cert major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9eb34322-124e-484d-a4fa-d79391ac36b1/volumes/kubernetes.io~projected/kube-api-access-zjdgx:{mountpoint:/var/lib/kubelet/pods/9eb34322-124e-484d-a4fa-d79391ac36b1/volumes/kubernetes.io~projected/kube-api-access-zjdgx major:0 minor:325 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9eb34322-124e-484d-a4fa-d79391ac36b1/volumes/kubernetes.io~secret/monitoring-plugin-cert:{mountpoint:/var/lib/kubelet/pods/9eb34322-124e-484d-a4fa-d79391ac36b1/volumes/kubernetes.io~secret/monitoring-plugin-cert major:0 minor:324 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/abdd79d9-8d87-4133-85a2-643d26fc9faa/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/abdd79d9-8d87-4133-85a2-643d26fc9faa/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/abdd79d9-8d87-4133-85a2-643d26fc9faa/volumes/kubernetes.io~projected/kube-api-access-h9j8g:{mountpoint:/var/lib/kubelet/pods/abdd79d9-8d87-4133-85a2-643d26fc9faa/volumes/kubernetes.io~projected/kube-api-access-h9j8g major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/abdd79d9-8d87-4133-85a2-643d26fc9faa/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/abdd79d9-8d87-4133-85a2-643d26fc9faa/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/acada406-fef9-4a03-ae21-c2351ef0a4cf/volumes/kubernetes.io~projected/kube-api-access-9rwtq:{mountpoint:/var/lib/kubelet/pods/acada406-fef9-4a03-ae21-c2351ef0a4cf/volumes/kubernetes.io~projected/kube-api-access-9rwtq major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bff481c2-d05e-4b45-827d-f5c8a0f60d0b/volumes/kubernetes.io~projected/kube-api-access-88c5l:{mountpoint:/var/lib/kubelet/pods/bff481c2-d05e-4b45-827d-f5c8a0f60d0b/volumes/kubernetes.io~projected/kube-api-access-88c5l major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d157814d-82d6-4af5-a648-c03ad35f3a41/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/d157814d-82d6-4af5-a648-c03ad35f3a41/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d157814d-82d6-4af5-a648-c03ad35f3a41/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/d157814d-82d6-4af5-a648-c03ad35f3a41/volumes/kubernetes.io~empty-dir/tmp major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d157814d-82d6-4af5-a648-c03ad35f3a41/volumes/kubernetes.io~projected/kube-api-access-cct5g:{mountpoint:/var/lib/kubelet/pods/d157814d-82d6-4af5-a648-c03ad35f3a41/volumes/kubernetes.io~projected/kube-api-access-cct5g major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e17d6c62-2952-420c-beda-49db831c48ac/volumes/kubernetes.io~projected/kube-api-access-g7skt:{mountpoint:/var/lib/kubelet/pods/e17d6c62-2952-420c-beda-49db831c48ac/volumes/kubernetes.io~projected/kube-api-access-g7skt major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fe434c5e-450b-4512-93aa-78d3e56e5002/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/fe434c5e-450b-4512-93aa-78d3e56e5002/volumes/kubernetes.io~secret/agent-certs major:0 minor:60 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/41db31f67562ce3fb085ff9b99bbe0be0bd1b469e27f4f97a805d4e6405a2d96/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/d40b4bbae2fb85e531c60fd2e4ddb3d5ab6b711da0cd0f61c42f37377b88ebf6/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/c02a2ba412cc56a9340d84853111fc5fa5b20f2190a25d8b3e8efcecb6f85d65/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/02932c6b9d5f7bbb772bfaeec3be178db66a45a62283b9d24ece9952d65e7832/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-111:{mountpoint:/var/lib/containers/storage/overlay/eab11ba705af3cd6873907121f858790728b7201f7a89f7ab12e2adf2f814ea3/merged major:0 minor:111 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/1881fe9fd42cc0e59a271aa7f21d593f490a8337b3906261519c7f63f92db9be/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/e1a5e87727123b9e05473e2bf10e568cc2b0ac766a53681c9d9e5230371bea4d/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/4b474afb704ed6757f79bf0283d75cbe2e367b7d50bd8b4dc20a532cab16bc4b/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/c894f937a22abbfd0353572dbe2b3d1ecf2b7d7d1f51bd82cd3ee40ca8ff0a57/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/8d95a91b8b72f3c953115ead2b2b09afaeadcdfa0392bb67bb0dad946d7fdd4b/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/d9dbf5a403e8e06caa2b56f5d41ccbdc182ce775537ec01afc98ad11014ebc4a/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/40156758f808ec27996768e43a2297aae44014c255b008a366ae70d1b70207e1/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/1c78745a199f3e40b0fbb8487091734a56f7fa4827b756662c44257715f1feed/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/a7078a6962940aa460f079eade9032fa888b9fa0f581ca51b6deb22cc2064f2e/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/77692cd8ccbb11d19f7417d4f4a256176844b02e8e7d938ca332af2c1350b2c4/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/381d2ee5be8672f46c95764eff64385a831fbc20207a4456768cf0808db61727/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/298c509c31b77fdf5f8c82300934a71715cb52c6f0cfa52037cedbf50a3c72a5/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/925ab3fe8a260f1381582a46192eed6fb4a297141594bf1a23b8d5a7e25b43a6/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/b4ce744c7953caef7d2be3da2cb9c7c11526b065afe35d2da46d6b9598bab9bf/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/a98e3ced74b1ceae15cd028c315e361d8b1bff191e6cc65d55983856cec06fee/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/7435c74c7fa0ccbc52863cd9c437aaab2f9d76a494d867f00d2af50596335db1/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/f70ba5c29e76dace564c534748e595a101e917e5dfd3e8a142bf25dd670d41d7/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/0fd1a0867ce518643c1b69c781a9820c61dc780fd089245d9d8cfea861882e79/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/d9440f6551a0fbfcb85cf731ce46cac24aed4791404786435a456222f9d34543/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/2daf59cb3aa0eefca6debc7d8cbec0d9624f45b5b4ecdbabf73a60f72a7eb5fe/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-219:{mountpoint:/var/lib/containers/storage/overlay/f1f21d17bc80655b9a6c419683f80f81abc7f56b0367670310e62e45f2aa400d/merged major:0 minor:219 fsType:overlay blockSize:0} overlay_0-225:{mountpoint:/var/lib/containers/storage/overlay/04b5db83d4ea1bd795f024ac7703deefbec325aa46718d269d2879b8a390692d/merged major:0 minor:225 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/842355e0a5b695a6ae3cd4fb82d0de734ab49eae4e1bfd855251ee3d17279157/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/765d7e639ccfb32efcf7221c50aabcc9770f3cccfc10f477e6a8e71c4057e840/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/379d17041c1475fe5d97ad4778070d3655623813e00db8f9c9c8d9534038cfb7/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/8d0dcce485958b3f00b39fd70238cd75bc92371497cc4761ea7ca69e62d8c38c/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/310a3598ef551908fff2cc531f062649c2f4ae21bdf8bbcd9bc6f215c61fc355/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/e8b26178c99d12cde791d77a8e0ccffdc754ddd36077554fb2e01fa0d4fbf886/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-272:{mountpoint:/var/lib/containers/storage/overlay/cd0ba58671de3fc01ad6eb9cf2e3201760a3f665857287f056f9b218182ca642/merged major:0 minor:272 fsType:overlay blockSize:0} overlay_0-274:{mountpoint:/var/lib/containers/storage/overlay/cfeb8d50e10a2a52ee31b1b38eb23bd5fa27cb1c0210bf208bb08f031053e629/merged major:0 minor:274 fsType:overlay blockSize:0} overlay_0-279:{mountpoint:/var/lib/containers/storage/overlay/d25338c2ec6369f170048dfccfeca02d3b188cd80924b12e286332f89396d98e/merged major:0 minor:279 fsType:overlay blockSize:0} overlay_0-284:{mountpoint:/var/lib/containers/storage/overlay/9a8f0af7ca6a6031a15d5848a064089e6b4c8738dec85a702cade3535035aa3d/merged major:0 minor:284 fsType:overlay blockSize:0} overlay_0-289:{mountpoint:/var/lib/containers/storage/overlay/95df43571327f182766fe9ca99653b7dddf1dcac21435c8c178e964118f21d3f/merged major:0 minor:289 fsType:overlay blockSize:0} overlay_0-306:{mountpoint:/var/lib/containers/storage/overlay/3f3e22fef31441940fbdf8ba09cadd38113bffba9ed1d5921f24d31eabd190cc/merged major:0 minor:306 fsType:overlay blockSize:0} overlay_0-308:{mountpoint:/var/lib/containers/storage/overlay/c2fa39f8334bee0a29e3c00ee982af86a3df6f4da2502abb78e0abf07758caa1/merged major:0 minor:308 fsType:overlay blockSize:0} overlay_0-314:{mountpoint:/var/lib/containers/storage/overlay/d83321c0b848699646e004ed623df3f6208efca68096f927b5d612ef4f414434/merged major:0 minor:314 fsType:overlay blockSize:0} overlay_0-319:{mountpoint:/var/lib/containers/storage/overlay/bcbf5220b3355dddc9b4b10c74518667f4590c1e751f5647daed928eadce38fa/merged major:0 minor:319 fsType:overlay blockSize:0} overlay_0-328:{mountpoint:/var/lib/containers/storage/overlay/08fc390f50f2fffc794f38fbaaec50b3d5e79240654a5daa65d2f687cd9dfbb7/merged major:0 minor:328 fsType:overlay blockSize:0} overlay_0-330:{mountpoint:/var/lib/containers/storage/overlay/ca896048be754ddea4c330df82df5bb6a84cb6d6a236f5c65455cedffbda414c/merged major:0 minor:330 fsType:overlay blockSize:0} overlay_0-332:{mountpoint:/var/lib/containers/storage/overlay/9f9cccf1560e0c5bd3443c945d1f8b84a0f56fa31b8b696982968508641ae566/merged major:0 minor:332 fsType:overlay blockSize:0} overlay_0-338:{mountpoint:/var/lib/containers/storage/overlay/a2d6de787d7ad03f21e2e499efa0cb49cb901960b01d7f6455f2b0d3ff5f28eb/merged major:0 minor:338 fsType:overlay blockSize:0} overlay_0-343:{mountpoint:/var/lib/containers/storage/overlay/3436afa8c88c03897697914e879e24a7db6fed2898ebab6968f12e1381b02b67/merged major:0 minor:343 fsType:overlay blockSize:0} overlay_0-357:{mountpoint:/var/lib/containers/storage/overlay/c53d70262de393f4e6512b0506ed1e62a98d9b8f315af762933017fd618974f3/merged major:0 minor:357 fsType:overlay blockSize:0} overlay_0-359:{mountpoint:/var/lib/containers/storage/overlay/f19673219440b3bc424dc851df0d766ac263cde0e79f0933fa2406e7bcdd6b94/merged major:0 minor:359 fsType:overlay blockSize:0} overlay_0-364:{mountpoint:/var/lib/containers/storage/overlay/7e548e70176a1f1c93949b1685de5abc3a8dc6795ea026ac507f306be88a1716/merged major:0 minor:364 fsType:overlay blockSize:0} overlay_0-366:{mountpoint:/var/lib/containers/storage/overlay/e9142c7eff1a878fed33b772ed591fc1007677272004dece5560f204325ed87e/merged major:0 minor:366 fsType:overlay blockSize:0} overlay_0-371:{mountpoint:/var/lib/containers/storage/overlay/b747e66fd316e3d7356c71cea9afa47c32e79eae225953ca2c655462999a7d8f/merged major:0 minor:371 fsType:overlay blockSize:0} overlay_0-389:{mountpoint:/var/lib/containers/storage/overlay/eacdb069ba4bd082e9b885a41e1b28ac5b0e8fef5c07555bbe22009dd1776b67/merged major:0 minor:389 fsType:overlay blockSize:0} overlay_0-391:{mountpoint:/var/lib/containers/storage/overlay/4182eb79c1c0f545733db051a6935fdfacd93719c98c6ad344c2fb1c60b9d83f/merged major:0 minor:391 fsType:overlay blockSize:0} overlay_0-393:{mountpoint:/var/lib/containers/storage/overlay/e0ab79a256d38114ed109745a125140921628f668c1c4f1546ee0143e1395640/merged major:0 minor:393 fsType:overlay blockSize:0} overlay_0-395:{mountpoint:/var/lib/containers/storage/overlay/2e2df4b53524a4ef9713b5263fed7d87958fbe23b5d9af1661dbd6747609c2d5/merged major:0 minor:395 fsType:overlay blockSize:0} overlay_0-397:{mountpoint:/var/lib/containers/storage/overlay/0ac4aeb30c6b4bf645e67f14d60d008828b7fff03e3a6d1670c00fafa7c01ff9/merged major:0 minor:397 fsType:overlay blockSize:0} overlay_0-403:{mountpoint:/var/lib/containers/storage/overlay/e178a53de975b21a7f8f274cc1e09739626dae44ebbfdb4287d5b3149929ed18/merged major:0 minor:403 fsType:overlay blockSize:0} overlay_0-408:{mountpoint:/var/lib/containers/storage/overlay/83bb5a5caa042784a0ab7420d2203d1cb7fa1e190a169c3046d5de4d2865f809/merged major:0 minor:408 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/45fcef7ea58dd092c3a2b7931ad302b996c67830c4755ce02419eafadc175194/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/ae7413f8780276a4310846ebe57f16464e427edaa72f49e826dccd6b0cc2bc25/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/cabf0725757f17dcdc3d5f9b4b8f7a9f4cf49d8ec3f9b9befbb46c0bc7e8dd36/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/028caebfe331ba1e95dbbfbaeb43b0e65776791c18f03bd2ff7f472aa0739d0b/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/b1950f9e387e062821bc560eb197a93ace3cd2f426392369f899cfa127fe9f63/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/33e9c13a80546b5d23c4506131cb6d1001d86bcd5ed2d0a897635c6d60cdf3fe/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/60147b6c50119a2697769a4ee94b5033b2fe8f9b9ae7d6a2f0a7175d2bc33fdd/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/b6e742feb50fc1af5a88952f09bdcff8c4a7d1592f243a8165af9673da464e33/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/0be88f1d66ad872dea34225ba3f4788822e47f4d65cdb057b4305f7907ca9c2a/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/888f0cbf6f726ced73ff79202d2cf79a4341c29c63a7353771e1c21fc1b984b3/merged major:0 minor:96 fsType:overlay blockSize:0}] Nov 03 01:18:38.030180 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.029758 5437 manager.go:217] Machine: {Timestamp:2026-11-03 01:18:38.029264494 +0000 UTC m=+0.058746780 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099878 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2737fedf27e71eb9003a4a4870e7a4 SystemUUID:ec2737fe-df27-e71e-b900-3a4a4870e7a4 BootID:fbdbed05-f2cf-42d7-91da-8abea231b5a2 Filesystems:[{Device:overlay_0-395 DeviceMajor:0 DeviceMinor:395 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d157814d-82d6-4af5-a648-c03ad35f3a41/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:58 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-328 DeviceMajor:0 DeviceMinor:328 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f1bbf393f14b3c0361a059eb428e4657c6a8c03af1b7fc3416c78ecb0a1423f1/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/97cb83aa-8922-4221-8407-fbbb6c39f5c7/volumes/kubernetes.io~projected/kube-api-access-4v9cn DeviceMajor:0 DeviceMinor:205 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/beaf53b14654cf966260cc8f6b640829a5f2cbd5414b1aa7902e5fb960ffd8fb/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-397 DeviceMajor:0 DeviceMinor:397 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4c25011b-c85c-4736-9f72-842a96949072/volumes/kubernetes.io~projected/kube-api-access-vzr84 DeviceMajor:0 DeviceMinor:64 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7c7251f3fe76aa5de4ec3a3488313d38429bb12e51be59710fb3d62113862144/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-274 DeviceMajor:0 DeviceMinor:274 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d157814d-82d6-4af5-a648-c03ad35f3a41/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:59 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/d157814d-82d6-4af5-a648-c03ad35f3a41/volumes/kubernetes.io~projected/kube-api-access-cct5g DeviceMajor:0 DeviceMinor:61 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/386397a9-5c8d-4221-894b-279e0b396afe/volumes/kubernetes.io~projected/kube-api-access-wk6dw DeviceMajor:0 DeviceMinor:354 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7e4535033e2187d3a887d160f43211e0b3f012c6b23c0b04d27c966b04e03c50/userdata/shm DeviceMajor:0 DeviceMinor:355 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-408 DeviceMajor:0 DeviceMinor:408 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/766a044f-fc66-4998-8006-343c0329860d/volumes/kubernetes.io~secret/kube-state-metrics-tls DeviceMajor:0 DeviceMinor:244 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/3b2bf2f6-28aa-4480-9f4a-f4e7f659e966/volumes/kubernetes.io~projected/kube-api-access-vgd8w DeviceMajor:0 DeviceMinor:382 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/24bd13d85cff8f0cf46be8ea2d05895b41b5b5beed80366bec3a1469068ed1ab/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/acada406-fef9-4a03-ae21-c2351ef0a4cf/volumes/kubernetes.io~projected/kube-api-access-9rwtq DeviceMajor:0 DeviceMinor:57 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3a98ca3c6966d3e44ce6dcd1ee13bd9cdf5b46bbeb0c34fe97865c015118ff10/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/406839857f2f8eaa16b794d2e388372be8131e40e4df221e5bea5891f1d3cd05/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8faf3e8cf655ef04445ca37b09e02a7a6303ffbe2ea86a0fd98a8e918838d88d/userdata/shm DeviceMajor:0 DeviceMinor:203 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a8e0819cc9cee4743891c1bd3a02cdedd551a431d84689742656f35cef6c34de/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0e35c4cd-ef18-40c1-8568-f77773e378b2/volumes/kubernetes.io~secret/openshift-state-metrics-tls DeviceMajor:0 DeviceMinor:236 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/766a044f-fc66-4998-8006-343c0329860d/volumes/kubernetes.io~projected/kube-api-access-b7ckb DeviceMajor:0 DeviceMinor:248 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/8f478179-5984-48dd-b872-8f552954ecea/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:298 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0e9e7edee56b546df74773696b5a5d39e39e5a667ed6d072684745e5f19334ff/userdata/shm DeviceMajor:0 DeviceMinor:251 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-306 DeviceMajor:0 DeviceMinor:306 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/30eede54-1970-4f52-b219-70d3c469ddd4/volumes/kubernetes.io~projected/kube-api-access-sq5fx DeviceMajor:0 DeviceMinor:249 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-366 DeviceMajor:0 DeviceMinor:366 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-308 DeviceMajor:0 DeviceMinor:308 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/20152fc3-0b5f-4c58-94fb-e02094377c74/volumes/kubernetes.io~projected/kube-api-access-m2r8s DeviceMajor:0 DeviceMinor:63 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/1febd8fa-5641-449b-b1fd-c7d02603fd1c/volumes/kubernetes.io~projected/kube-api-access-bdm2b DeviceMajor:0 DeviceMinor:62 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/24a6b7dfc849cfef3310febb8cba625fb2635841a98b3907ddfe9a3951c334ba/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1fdb6f2d-9138-4f57-af12-c3909dff6e5f/volumes/kubernetes.io~projected/kube-api-access-8flcc DeviceMajor:0 DeviceMinor:197 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/8f478179-5984-48dd-b872-8f552954ecea/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules DeviceMajor:0 DeviceMinor:301 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/9eb34322-124e-484d-a4fa-d79391ac36b1/volumes/kubernetes.io~secret/monitoring-plugin-cert DeviceMajor:0 DeviceMinor:324 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/8f478179-5984-48dd-b872-8f552954ecea/volumes/kubernetes.io~projected/kube-api-access-v8l47 DeviceMajor:0 DeviceMinor:303 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6ecc2615-b458-41f8-839f-e22c53e7ad0e/volumes/kubernetes.io~projected/kube-api-access-b56kh DeviceMajor:0 DeviceMinor:65 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-111 DeviceMajor:0 DeviceMinor:111 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-279 DeviceMajor:0 DeviceMinor:279 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8f478179-5984-48dd-b872-8f552954ecea/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy DeviceMajor:0 DeviceMinor:294 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/9eb34322-124e-484d-a4fa-d79391ac36b1/volumes/kubernetes.io~projected/kube-api-access-zjdgx DeviceMajor:0 DeviceMinor:325 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/e17d6c62-2952-420c-beda-49db831c48ac/volumes/kubernetes.io~projected/kube-api-access-g7skt DeviceMajor:0 DeviceMinor:66 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/abdd79d9-8d87-4133-85a2-643d26fc9faa/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-272 DeviceMajor:0 DeviceMinor:272 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6ecc2615-b458-41f8-839f-e22c53e7ad0e/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:376 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/6179266f-07ff-48cd-bf03-3f1e961529d7/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:381 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-391 DeviceMajor:0 DeviceMinor:391 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1fdb6f2d-9138-4f57-af12-c3909dff6e5f/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:45 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a0a9ec2bdcdd88a98fc92b6fa9724aac0472c842079f5b018614d7d35bef61a2/userdata/shm DeviceMajor:0 DeviceMinor:250 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-357 DeviceMajor:0 DeviceMinor:357 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2320cce9-0fdb-470a-ae01-e88118ba0419/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:201 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/0e35c4cd-ef18-40c1-8568-f77773e378b2/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:240 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-389 DeviceMajor:0 DeviceMinor:389 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/abdd79d9-8d87-4133-85a2-643d26fc9faa/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:55 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-319 DeviceMajor:0 DeviceMinor:319 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3718cadbf973e67ccd8645ba1efff38c7704ba18c1a9ca4d2d3c853a7544a95d/userdata/shm DeviceMajor:0 DeviceMinor:386 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b47a157f9083e86acb094f9f6f448c186b115df3f282b88bae7bb034ef72ee4e/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8f478179-5984-48dd-b872-8f552954ecea/volumes/kubernetes.io~secret/secret-thanos-querier-tls DeviceMajor:0 DeviceMinor:300 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/8f478179-5984-48dd-b872-8f552954ecea/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics DeviceMajor:0 DeviceMinor:302 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/386397a9-5c8d-4221-894b-279e0b396afe/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:348 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/0e35c4cd-ef18-40c1-8568-f77773e378b2/volumes/kubernetes.io~projected/kube-api-access-shwrh DeviceMajor:0 DeviceMinor:241 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/30eede54-1970-4f52-b219-70d3c469ddd4/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:246 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-332 DeviceMajor:0 DeviceMinor:332 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-338 DeviceMajor:0 DeviceMinor:338 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/386397a9-5c8d-4221-894b-279e0b396afe/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:353 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a57cfab1072e0e4b06e4b4217747174d1f2639b037b9403811af934a57dc23e5/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/abdd79d9-8d87-4133-85a2-643d26fc9faa/volumes/kubernetes.io~projected/kube-api-access-h9j8g DeviceMajor:0 DeviceMinor:56 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1e4ab311b886828acac64bcbb2ec9807a19f1a8f64a2faaf1ba578edef09bfd1/userdata/shm DeviceMajor:0 DeviceMinor:385 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4d1338fb1357347638c76068111fb2982b81469361736cb026758c7679f468a2/userdata/shm DeviceMajor:0 DeviceMinor:304 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-359 DeviceMajor:0 DeviceMinor:359 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/97cb83aa-8922-4221-8407-fbbb6c39f5c7/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:202 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/97a126238281ad8365de918d31b44613980842c7d124f305133855532128b7c3/userdata/shm DeviceMajor:0 DeviceMinor:326 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-343 DeviceMajor:0 DeviceMinor:343 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-371 DeviceMajor:0 DeviceMinor:371 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-403 DeviceMajor:0 DeviceMinor:403 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ce3ef08fafa3dfc7c74905d661a123ef822c6af06a08d3f565ed2340bed81e4a/userdata/shm DeviceMajor:0 DeviceMinor:383 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/640208824d01bacb81bae87037315951d0d6301e18043589626cbb13a7a78d2b/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-330 DeviceMajor:0 DeviceMinor:330 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bff481c2-d05e-4b45-827d-f5c8a0f60d0b/volumes/kubernetes.io~projected/kube-api-access-88c5l DeviceMajor:0 DeviceMinor:67 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/31db35db8133fd4c1b3f0909bcef183472dca46c17fb140c4240f9c733b82412/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-219 DeviceMajor:0 DeviceMinor:219 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/766a044f-fc66-4998-8006-343c0329860d/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:245 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-393 DeviceMajor:0 DeviceMinor:393 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fe434c5e-450b-4512-93aa-78d3e56e5002/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:60 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-225 DeviceMajor:0 DeviceMinor:225 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/30eede54-1970-4f52-b219-70d3c469ddd4/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:247 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-284 DeviceMajor:0 DeviceMinor:284 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-289 DeviceMajor:0 DeviceMinor:289 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ec11b93955fdbcf32dfeefb15d657d495d24cb563f8d5b07067303a2cc67c40a/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2320cce9-0fdb-470a-ae01-e88118ba0419/volumes/kubernetes.io~projected/kube-api-access-b2cz9 DeviceMajor:0 DeviceMinor:206 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/8f478179-5984-48dd-b872-8f552954ecea/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:299 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-314 DeviceMajor:0 DeviceMinor:314 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-364 DeviceMajor:0 DeviceMinor:364 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:1e4ab311b886828 MacAddress:d2:19:fa:6a:ec:06 Speed:10000 Mtu:8901} {Name:3718cadbf973e67 MacAddress:0a:23:b6:58:9f:a5 Speed:10000 Mtu:8901} {Name:4d1338fb1357347 MacAddress:4a:3d:4c:9e:c8:4e Speed:10000 Mtu:8901} {Name:7c7251f3fe76aa5 MacAddress:7a:a2:6c:e4:42:05 Speed:10000 Mtu:8901} {Name:7e4535033e2187d MacAddress:26:a0:f9:ab:48:5e Speed:10000 Mtu:8901} {Name:8faf3e8cf655ef0 MacAddress:da:19:30:5d:85:31 Speed:10000 Mtu:8901} {Name:97a126238281ad8 MacAddress:52:af:8f:db:d6:20 Speed:10000 Mtu:8901} {Name:a0a9ec2bdcdd88a MacAddress:fa:a4:a8:b7:7e:fc Speed:10000 Mtu:8901} {Name:a8e0819cc9cee47 MacAddress:da:d4:0b:00:ca:3a Speed:10000 Mtu:8901} {Name:beaf53b14654cf9 MacAddress:d6:ce:98:16:e3:d2 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:06:ff:ca:24:12:bb Speed:0 Mtu:9001} {Name:br-int MacAddress:8a:83:74:77:4a:f1 Speed:0 Mtu:8901} {Name:ce3ef08fafa3dfc MacAddress:12:47:d0:d0:1e:9d Speed:10000 Mtu:8901} {Name:ens5 MacAddress:06:ff:ca:24:12:bb Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:0a:dc:ee:1e:85:44 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:f6:a6:c1:38:f3:6a Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 03 01:18:38.030355 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.030184 5437 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 03 01:18:38.030355 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.030234 5437 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 03 01:18:38.030572 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.030543 5437 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 03 01:18:38.030701 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.030574 5437 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-16.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 03 01:18:38.030742 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.030704 5437 topology_manager.go:143] "Creating topology manager with none policy" Nov 03 01:18:38.030742 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.030712 5437 container_manager_linux.go:311] "Creating device plugin manager" Nov 03 01:18:38.030742 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.030723 5437 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 01:18:38.030818 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.030742 5437 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 01:18:38.030849 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.030840 5437 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 03 01:18:38.031027 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.031017 5437 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 03 01:18:38.031128 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.031120 5437 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 03 01:18:38.031159 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.031155 5437 kubelet.go:493] "Attempting to sync node with API server" Nov 03 01:18:38.031189 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.031168 5437 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 03 01:18:38.031189 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.031179 5437 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 03 01:18:38.031189 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.031188 5437 kubelet.go:405] "Adding apiserver pod source" Nov 03 01:18:38.031265 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.031196 5437 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 03 01:18:38.032154 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.032137 5437 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 03 01:18:38.032154 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.032153 5437 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 01:18:38.032787 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.032767 5437 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 03 01:18:38.035871 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.035853 5437 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 03 01:18:38.035871 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.035869 5437 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 01:18:38.036020 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.035994 5437 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 03 01:18:38.036246 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.036230 5437 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 03 01:18:38.036310 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.036271 5437 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 03 01:18:38.036454 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.036443 5437 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 03 01:18:38.036522 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.036463 5437 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 03 01:18:38.036522 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.036474 5437 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 03 01:18:38.036522 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.036483 5437 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 03 01:18:38.036522 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.036492 5437 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 03 01:18:38.036522 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.036502 5437 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 03 01:18:38.036522 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.036511 5437 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 03 01:18:38.036522 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.036520 5437 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 03 01:18:38.036834 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.036531 5437 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 03 01:18:38.036834 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.036540 5437 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 03 01:18:38.036834 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.036553 5437 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 03 01:18:38.036834 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.036568 5437 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 03 01:18:38.036834 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.036605 5437 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 03 01:18:38.036834 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.036615 5437 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 03 01:18:38.040153 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.040135 5437 server.go:1263] "Started kubelet" Nov 03 01:18:38.040258 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.040228 5437 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 03 01:18:38.047874 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.040374 5437 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 03 01:18:38.047874 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.041126 5437 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 03 01:18:38.047874 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.042069 5437 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 03 01:18:38.047874 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.043722 5437 server.go:317] "Adding debug handlers to kubelet server" Nov 03 01:18:38.047874 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.046290 5437 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 03 01:18:38.047874 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.046307 5437 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 03 01:18:38.047874 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.046412 5437 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 01:12:26 +0000 UTC" deadline="2028-04-24 07:43:27.461864482 +0000 UTC" Nov 03 01:18:38.047874 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.046460 5437 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12918h24m49.415408999s" Nov 03 01:18:38.047874 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.046745 5437 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 03 01:18:38.047874 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.046758 5437 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 03 01:18:38.047874 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.046947 5437 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 03 01:18:38.047874 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:18:38.047052 5437 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-16.us-west-2.compute.internal\" not found" Nov 03 01:18:38.041472 ip-10-0-136-16 systemd[1]: Started Kubernetes Kubelet. Nov 03 01:18:38.048641 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.048554 5437 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 03 01:18:38.048641 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.048570 5437 factory.go:55] Registering systemd factory Nov 03 01:18:38.048641 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.048582 5437 factory.go:223] Registration of the systemd container factory successfully Nov 03 01:18:38.052309 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.049014 5437 factory.go:153] Registering CRI-O factory Nov 03 01:18:38.052309 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.049038 5437 factory.go:223] Registration of the crio container factory successfully Nov 03 01:18:38.052309 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.049057 5437 factory.go:103] Registering Raw factory Nov 03 01:18:38.052309 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.049071 5437 manager.go:1199] Started watching for new ooms in manager Nov 03 01:18:38.052309 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.049517 5437 manager.go:319] Starting recovery of all containers Nov 03 01:18:38.052309 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.049629 5437 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2320cce9-0fdb-470a-ae01-e88118ba0419" volumeName="kubernetes.io/projected/2320cce9-0fdb-470a-ae01-e88118ba0419-kube-api-access-b2cz9" seLinuxMountContext="" Nov 03 01:18:38.052309 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.049666 5437 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="386397a9-5c8d-4221-894b-279e0b396afe" volumeName="kubernetes.io/configmap/386397a9-5c8d-4221-894b-279e0b396afe-trusted-ca-bundle" seLinuxMountContext="" Nov 03 01:18:38.052309 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.049676 5437 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4c25011b-c85c-4736-9f72-842a96949072" volumeName="kubernetes.io/configmap/4c25011b-c85c-4736-9f72-842a96949072-cni-binary-copy" seLinuxMountContext="" Nov 03 01:18:38.052309 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.049685 5437 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6ecc2615-b458-41f8-839f-e22c53e7ad0e" volumeName="kubernetes.io/secret/6ecc2615-b458-41f8-839f-e22c53e7ad0e-metrics-certs" seLinuxMountContext="" Nov 03 01:18:38.052309 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.049694 5437 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="766a044f-fc66-4998-8006-343c0329860d" volumeName="kubernetes.io/empty-dir/766a044f-fc66-4998-8006-343c0329860d-volume-directive-shadow" seLinuxMountContext="" Nov 03 01:18:38.052309 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.049703 5437 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d157814d-82d6-4af5-a648-c03ad35f3a41" volumeName="kubernetes.io/empty-dir/d157814d-82d6-4af5-a648-c03ad35f3a41-etc-tuned" seLinuxMountContext="" Nov 03 01:18:38.052309 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.049711 5437 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="20152fc3-0b5f-4c58-94fb-e02094377c74" volumeName="kubernetes.io/projected/20152fc3-0b5f-4c58-94fb-e02094377c74-kube-api-access-m2r8s" seLinuxMountContext="" Nov 03 01:18:38.052309 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.049718 5437 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2320cce9-0fdb-470a-ae01-e88118ba0419" volumeName="kubernetes.io/secret/2320cce9-0fdb-470a-ae01-e88118ba0419-insights-runtime-extractor-tls" seLinuxMountContext="" Nov 03 01:18:38.052309 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.049734 5437 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="766a044f-fc66-4998-8006-343c0329860d" volumeName="kubernetes.io/configmap/766a044f-fc66-4998-8006-343c0329860d-metrics-client-ca" seLinuxMountContext="" Nov 03 01:18:38.052309 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.049747 5437 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8f478179-5984-48dd-b872-8f552954ecea" volumeName="kubernetes.io/secret/8f478179-5984-48dd-b872-8f552954ecea-secret-thanos-querier-kube-rbac-proxy-web" seLinuxMountContext="" Nov 03 01:18:38.052309 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.049755 5437 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="abdd79d9-8d87-4133-85a2-643d26fc9faa" volumeName="kubernetes.io/secret/abdd79d9-8d87-4133-85a2-643d26fc9faa-ovn-node-metrics-cert" seLinuxMountContext="" Nov 03 01:18:38.052309 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.049762 5437 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e17d6c62-2952-420c-beda-49db831c48ac" volumeName="kubernetes.io/configmap/e17d6c62-2952-420c-beda-49db831c48ac-cni-binary-copy" seLinuxMountContext="" Nov 03 01:18:38.052309 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.049771 5437 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0e35c4cd-ef18-40c1-8568-f77773e378b2" volumeName="kubernetes.io/secret/0e35c4cd-ef18-40c1-8568-f77773e378b2-openshift-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Nov 03 01:18:38.052309 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.049778 5437 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1fdb6f2d-9138-4f57-af12-c3909dff6e5f" volumeName="kubernetes.io/projected/1fdb6f2d-9138-4f57-af12-c3909dff6e5f-kube-api-access-8flcc" seLinuxMountContext="" Nov 03 01:18:38.052309 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.049787 5437 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="386397a9-5c8d-4221-894b-279e0b396afe" volumeName="kubernetes.io/configmap/386397a9-5c8d-4221-894b-279e0b396afe-console-config" seLinuxMountContext="" Nov 03 01:18:38.052309 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.049799 5437 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="abdd79d9-8d87-4133-85a2-643d26fc9faa" volumeName="kubernetes.io/projected/abdd79d9-8d87-4133-85a2-643d26fc9faa-kube-api-access-h9j8g" seLinuxMountContext="" Nov 03 01:18:38.052309 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.049815 5437 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="30eede54-1970-4f52-b219-70d3c469ddd4" volumeName="kubernetes.io/projected/30eede54-1970-4f52-b219-70d3c469ddd4-kube-api-access-sq5fx" seLinuxMountContext="" Nov 03 01:18:38.052309 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.049822 5437 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="386397a9-5c8d-4221-894b-279e0b396afe" volumeName="kubernetes.io/configmap/386397a9-5c8d-4221-894b-279e0b396afe-oauth-serving-cert" seLinuxMountContext="" Nov 03 01:18:38.052309 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.049829 5437 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="386397a9-5c8d-4221-894b-279e0b396afe" volumeName="kubernetes.io/secret/386397a9-5c8d-4221-894b-279e0b396afe-console-oauth-config" seLinuxMountContext="" Nov 03 01:18:38.052309 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.049836 5437 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8f478179-5984-48dd-b872-8f552954ecea" volumeName="kubernetes.io/configmap/8f478179-5984-48dd-b872-8f552954ecea-metrics-client-ca" seLinuxMountContext="" Nov 03 01:18:38.052309 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.049844 5437 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8f478179-5984-48dd-b872-8f552954ecea" volumeName="kubernetes.io/projected/8f478179-5984-48dd-b872-8f552954ecea-kube-api-access-v8l47" seLinuxMountContext="" Nov 03 01:18:38.052309 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.049852 5437 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="97cb83aa-8922-4221-8407-fbbb6c39f5c7" volumeName="kubernetes.io/secret/97cb83aa-8922-4221-8407-fbbb6c39f5c7-cert" seLinuxMountContext="" Nov 03 01:18:38.052309 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.049865 5437 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="386397a9-5c8d-4221-894b-279e0b396afe" volumeName="kubernetes.io/secret/386397a9-5c8d-4221-894b-279e0b396afe-console-serving-cert" seLinuxMountContext="" Nov 03 01:18:38.052309 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.049873 5437 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8f478179-5984-48dd-b872-8f552954ecea" volumeName="kubernetes.io/empty-dir/8f478179-5984-48dd-b872-8f552954ecea-tmp" seLinuxMountContext="" Nov 03 01:18:38.052309 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.049880 5437 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8f478179-5984-48dd-b872-8f552954ecea" volumeName="kubernetes.io/secret/8f478179-5984-48dd-b872-8f552954ecea-secret-thanos-querier-kube-rbac-proxy-metrics" seLinuxMountContext="" Nov 03 01:18:38.052309 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.049889 5437 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8f478179-5984-48dd-b872-8f552954ecea" volumeName="kubernetes.io/secret/8f478179-5984-48dd-b872-8f552954ecea-secret-grpc-tls" seLinuxMountContext="" Nov 03 01:18:38.052309 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.049896 5437 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d157814d-82d6-4af5-a648-c03ad35f3a41" volumeName="kubernetes.io/empty-dir/d157814d-82d6-4af5-a648-c03ad35f3a41-tmp" seLinuxMountContext="" Nov 03 01:18:38.052309 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.049905 5437 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0e35c4cd-ef18-40c1-8568-f77773e378b2" volumeName="kubernetes.io/configmap/0e35c4cd-ef18-40c1-8568-f77773e378b2-metrics-client-ca" seLinuxMountContext="" Nov 03 01:18:38.052309 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.049916 5437 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2320cce9-0fdb-470a-ae01-e88118ba0419" volumeName="kubernetes.io/configmap/2320cce9-0fdb-470a-ae01-e88118ba0419-kube-rbac-proxy-cm" seLinuxMountContext="" Nov 03 01:18:38.052309 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.049923 5437 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="30eede54-1970-4f52-b219-70d3c469ddd4" volumeName="kubernetes.io/secret/30eede54-1970-4f52-b219-70d3c469ddd4-node-exporter-tls" seLinuxMountContext="" Nov 03 01:18:38.052309 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.049930 5437 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="386397a9-5c8d-4221-894b-279e0b396afe" volumeName="kubernetes.io/projected/386397a9-5c8d-4221-894b-279e0b396afe-kube-api-access-wk6dw" seLinuxMountContext="" Nov 03 01:18:38.052309 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.049938 5437 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e17d6c62-2952-420c-beda-49db831c48ac" volumeName="kubernetes.io/configmap/e17d6c62-2952-420c-beda-49db831c48ac-multus-daemon-config" seLinuxMountContext="" Nov 03 01:18:38.052309 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.049945 5437 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1fdb6f2d-9138-4f57-af12-c3909dff6e5f" volumeName="kubernetes.io/empty-dir/1fdb6f2d-9138-4f57-af12-c3909dff6e5f-tmp-dir" seLinuxMountContext="" Nov 03 01:18:38.052309 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.049953 5437 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="30eede54-1970-4f52-b219-70d3c469ddd4" volumeName="kubernetes.io/configmap/30eede54-1970-4f52-b219-70d3c469ddd4-node-exporter-accelerators-collector-config" seLinuxMountContext="" Nov 03 01:18:38.052309 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.049960 5437 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="abdd79d9-8d87-4133-85a2-643d26fc9faa" volumeName="kubernetes.io/configmap/abdd79d9-8d87-4133-85a2-643d26fc9faa-ovnkube-config" seLinuxMountContext="" Nov 03 01:18:38.052309 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.049968 5437 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bff481c2-d05e-4b45-827d-f5c8a0f60d0b" volumeName="kubernetes.io/configmap/bff481c2-d05e-4b45-827d-f5c8a0f60d0b-serviceca" seLinuxMountContext="" Nov 03 01:18:38.052309 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.049982 5437 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4c25011b-c85c-4736-9f72-842a96949072" volumeName="kubernetes.io/projected/4c25011b-c85c-4736-9f72-842a96949072-kube-api-access-vzr84" seLinuxMountContext="" Nov 03 01:18:38.052309 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.049989 5437 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e17d6c62-2952-420c-beda-49db831c48ac" volumeName="kubernetes.io/projected/e17d6c62-2952-420c-beda-49db831c48ac-kube-api-access-g7skt" seLinuxMountContext="" Nov 03 01:18:38.052309 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.049996 5437 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fe434c5e-450b-4512-93aa-78d3e56e5002" volumeName="kubernetes.io/configmap/fe434c5e-450b-4512-93aa-78d3e56e5002-konnectivity-ca" seLinuxMountContext="" Nov 03 01:18:38.052309 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.050004 5437 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="766a044f-fc66-4998-8006-343c0329860d" volumeName="kubernetes.io/configmap/766a044f-fc66-4998-8006-343c0329860d-kube-state-metrics-custom-resource-state-configmap" seLinuxMountContext="" Nov 03 01:18:38.052309 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.050012 5437 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="766a044f-fc66-4998-8006-343c0329860d" volumeName="kubernetes.io/secret/766a044f-fc66-4998-8006-343c0329860d-kube-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Nov 03 01:18:38.052309 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.050019 5437 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8f478179-5984-48dd-b872-8f552954ecea" volumeName="kubernetes.io/secret/8f478179-5984-48dd-b872-8f552954ecea-secret-thanos-querier-kube-rbac-proxy-rules" seLinuxMountContext="" Nov 03 01:18:38.052309 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.050028 5437 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="20152fc3-0b5f-4c58-94fb-e02094377c74" volumeName="kubernetes.io/configmap/20152fc3-0b5f-4c58-94fb-e02094377c74-iptables-alerter-script" seLinuxMountContext="" Nov 03 01:18:38.052309 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.050035 5437 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8f478179-5984-48dd-b872-8f552954ecea" volumeName="kubernetes.io/secret/8f478179-5984-48dd-b872-8f552954ecea-secret-thanos-querier-tls" seLinuxMountContext="" Nov 03 01:18:38.052309 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.050044 5437 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="acada406-fef9-4a03-ae21-c2351ef0a4cf" volumeName="kubernetes.io/empty-dir/acada406-fef9-4a03-ae21-c2351ef0a4cf-tmp-dir" seLinuxMountContext="" Nov 03 01:18:38.052309 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.050056 5437 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1fdb6f2d-9138-4f57-af12-c3909dff6e5f" volumeName="kubernetes.io/secret/1fdb6f2d-9138-4f57-af12-c3909dff6e5f-metrics-tls" seLinuxMountContext="" Nov 03 01:18:38.052309 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.050067 5437 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4c25011b-c85c-4736-9f72-842a96949072" volumeName="kubernetes.io/configmap/4c25011b-c85c-4736-9f72-842a96949072-cni-sysctl-allowlist" seLinuxMountContext="" Nov 03 01:18:38.052309 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.050075 5437 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="766a044f-fc66-4998-8006-343c0329860d" volumeName="kubernetes.io/projected/766a044f-fc66-4998-8006-343c0329860d-kube-api-access-b7ckb" seLinuxMountContext="" Nov 03 01:18:38.052309 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.050083 5437 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="97cb83aa-8922-4221-8407-fbbb6c39f5c7" volumeName="kubernetes.io/projected/97cb83aa-8922-4221-8407-fbbb6c39f5c7-kube-api-access-4v9cn" seLinuxMountContext="" Nov 03 01:18:38.052309 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.050110 5437 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9eb34322-124e-484d-a4fa-d79391ac36b1" volumeName="kubernetes.io/secret/9eb34322-124e-484d-a4fa-d79391ac36b1-monitoring-plugin-cert" seLinuxMountContext="" Nov 03 01:18:38.052309 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.050122 5437 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d157814d-82d6-4af5-a648-c03ad35f3a41" volumeName="kubernetes.io/projected/d157814d-82d6-4af5-a648-c03ad35f3a41-kube-api-access-cct5g" seLinuxMountContext="" Nov 03 01:18:38.052309 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.050137 5437 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1fdb6f2d-9138-4f57-af12-c3909dff6e5f" volumeName="kubernetes.io/configmap/1fdb6f2d-9138-4f57-af12-c3909dff6e5f-config-volume" seLinuxMountContext="" Nov 03 01:18:38.052309 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.050148 5437 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1febd8fa-5641-449b-b1fd-c7d02603fd1c" volumeName="kubernetes.io/projected/1febd8fa-5641-449b-b1fd-c7d02603fd1c-kube-api-access-bdm2b" seLinuxMountContext="" Nov 03 01:18:38.052309 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.050162 5437 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="30eede54-1970-4f52-b219-70d3c469ddd4" volumeName="kubernetes.io/secret/30eede54-1970-4f52-b219-70d3c469ddd4-node-exporter-kube-rbac-proxy-config" seLinuxMountContext="" Nov 03 01:18:38.052309 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.050173 5437 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="386397a9-5c8d-4221-894b-279e0b396afe" volumeName="kubernetes.io/empty-dir/386397a9-5c8d-4221-894b-279e0b396afe-tmp" seLinuxMountContext="" Nov 03 01:18:38.052309 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.050182 5437 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="766a044f-fc66-4998-8006-343c0329860d" volumeName="kubernetes.io/secret/766a044f-fc66-4998-8006-343c0329860d-kube-state-metrics-tls" seLinuxMountContext="" Nov 03 01:18:38.052309 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.050198 5437 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9eb34322-124e-484d-a4fa-d79391ac36b1" volumeName="kubernetes.io/projected/9eb34322-124e-484d-a4fa-d79391ac36b1-kube-api-access-zjdgx" seLinuxMountContext="" Nov 03 01:18:38.052309 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.050211 5437 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="abdd79d9-8d87-4133-85a2-643d26fc9faa" volumeName="kubernetes.io/configmap/abdd79d9-8d87-4133-85a2-643d26fc9faa-ovnkube-script-lib" seLinuxMountContext="" Nov 03 01:18:38.052309 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.050222 5437 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="acada406-fef9-4a03-ae21-c2351ef0a4cf" volumeName="kubernetes.io/projected/acada406-fef9-4a03-ae21-c2351ef0a4cf-kube-api-access-9rwtq" seLinuxMountContext="" Nov 03 01:18:38.052309 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.050232 5437 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fe434c5e-450b-4512-93aa-78d3e56e5002" volumeName="kubernetes.io/secret/fe434c5e-450b-4512-93aa-78d3e56e5002-agent-certs" seLinuxMountContext="" Nov 03 01:18:38.052309 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.050243 5437 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0e35c4cd-ef18-40c1-8568-f77773e378b2" volumeName="kubernetes.io/secret/0e35c4cd-ef18-40c1-8568-f77773e378b2-openshift-state-metrics-tls" seLinuxMountContext="" Nov 03 01:18:38.052309 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.050254 5437 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="30eede54-1970-4f52-b219-70d3c469ddd4" volumeName="kubernetes.io/configmap/30eede54-1970-4f52-b219-70d3c469ddd4-metrics-client-ca" seLinuxMountContext="" Nov 03 01:18:38.052309 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.050266 5437 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4c25011b-c85c-4736-9f72-842a96949072" volumeName="kubernetes.io/configmap/4c25011b-c85c-4736-9f72-842a96949072-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 03 01:18:38.052309 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.050275 5437 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="30eede54-1970-4f52-b219-70d3c469ddd4" volumeName="kubernetes.io/empty-dir/30eede54-1970-4f52-b219-70d3c469ddd4-node-exporter-textfile" seLinuxMountContext="" Nov 03 01:18:38.052309 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.050315 5437 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3b2bf2f6-28aa-4480-9f4a-f4e7f659e966" volumeName="kubernetes.io/projected/3b2bf2f6-28aa-4480-9f4a-f4e7f659e966-kube-api-access-vgd8w" seLinuxMountContext="" Nov 03 01:18:38.052309 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.050332 5437 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6ecc2615-b458-41f8-839f-e22c53e7ad0e" volumeName="kubernetes.io/projected/6ecc2615-b458-41f8-839f-e22c53e7ad0e-kube-api-access-b56kh" seLinuxMountContext="" Nov 03 01:18:38.052309 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.050344 5437 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="abdd79d9-8d87-4133-85a2-643d26fc9faa" volumeName="kubernetes.io/configmap/abdd79d9-8d87-4133-85a2-643d26fc9faa-env-overrides" seLinuxMountContext="" Nov 03 01:18:38.052309 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.050356 5437 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0e35c4cd-ef18-40c1-8568-f77773e378b2" volumeName="kubernetes.io/projected/0e35c4cd-ef18-40c1-8568-f77773e378b2-kube-api-access-shwrh" seLinuxMountContext="" Nov 03 01:18:38.052309 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.050378 5437 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2320cce9-0fdb-470a-ae01-e88118ba0419" volumeName="kubernetes.io/empty-dir/2320cce9-0fdb-470a-ae01-e88118ba0419-data-volume" seLinuxMountContext="" Nov 03 01:18:38.052309 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.050391 5437 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="386397a9-5c8d-4221-894b-279e0b396afe" volumeName="kubernetes.io/configmap/386397a9-5c8d-4221-894b-279e0b396afe-service-ca" seLinuxMountContext="" Nov 03 01:18:38.052309 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.050401 5437 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6179266f-07ff-48cd-bf03-3f1e961529d7" volumeName="kubernetes.io/secret/6179266f-07ff-48cd-bf03-3f1e961529d7-original-pull-secret" seLinuxMountContext="" Nov 03 01:18:38.052309 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.050410 5437 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8f478179-5984-48dd-b872-8f552954ecea" volumeName="kubernetes.io/secret/8f478179-5984-48dd-b872-8f552954ecea-secret-thanos-querier-kube-rbac-proxy" seLinuxMountContext="" Nov 03 01:18:38.052309 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.050422 5437 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bff481c2-d05e-4b45-827d-f5c8a0f60d0b" volumeName="kubernetes.io/projected/bff481c2-d05e-4b45-827d-f5c8a0f60d0b-kube-api-access-88c5l" seLinuxMountContext="" Nov 03 01:18:38.052309 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.050431 5437 reconstruct.go:97] "Volume reconstruction finished" Nov 03 01:18:38.052309 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.050436 5437 reconciler.go:29] "Reconciler: start to sync state" Nov 03 01:18:38.079007 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.078956 5437 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 03 01:18:38.080840 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.080693 5437 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 03 01:18:38.080840 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.080719 5437 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 03 01:18:38.080840 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.080740 5437 kubelet.go:2521] "Starting kubelet main sync loop" Nov 03 01:18:38.080840 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:18:38.080781 5437 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 03 01:18:38.111573 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.111543 5437 generic.go:356] "Generic (PLEG): container finished" podID="4c25011b-c85c-4736-9f72-842a96949072" containerID="707a60ffd5841e43a67ccd194a43baf64a156b90613edd5247d050faf9033b68" exitCode=0 Nov 03 01:18:38.111573 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.111569 5437 generic.go:356] "Generic (PLEG): container finished" podID="4c25011b-c85c-4736-9f72-842a96949072" containerID="6cdb7dd6dda109a84fc562fdbbdb82f57b1618c0f25acc2e5f4ee17d4257a65b" exitCode=0 Nov 03 01:18:38.111573 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.111575 5437 generic.go:356] "Generic (PLEG): container finished" podID="4c25011b-c85c-4736-9f72-842a96949072" containerID="f2d3817df2b9d6350149ec0853e36b2981b85e189e92d129dd414d56096fa1c9" exitCode=0 Nov 03 01:18:38.111573 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.111581 5437 generic.go:356] "Generic (PLEG): container finished" podID="4c25011b-c85c-4736-9f72-842a96949072" containerID="10d60ccdb372fc5c82899df0ff62f14f2c97e4a68cee4802f08bf91d025ae9e8" exitCode=0 Nov 03 01:18:38.111808 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.111586 5437 generic.go:356] "Generic (PLEG): container finished" podID="4c25011b-c85c-4736-9f72-842a96949072" containerID="d9ae2a77a51cd08fc6eafd5480b933293ed28b4b72b4207ec38186ac55d18245" exitCode=0 Nov 03 01:18:38.111808 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.111591 5437 generic.go:356] "Generic (PLEG): container finished" podID="4c25011b-c85c-4736-9f72-842a96949072" containerID="6ff76455930533288223388e53328c54a4250811df4749a54a19a17c3c177f6f" exitCode=0 Nov 03 01:18:38.113491 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.113464 5437 generic.go:356] "Generic (PLEG): container finished" podID="f7e070a0c923c611d3cf96ab59dc8b57" containerID="6fb6612237ca18debdc4ee17b23d29d97c5806aea283b7663f3d2636bc452fbf" exitCode=0 Nov 03 01:18:38.118714 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.118682 5437 generic.go:356] "Generic (PLEG): container finished" podID="30eede54-1970-4f52-b219-70d3c469ddd4" containerID="569da36b752ded8ec2d1f5e3ab6261cebfb8cc955d1e71d8d2b15cf031367b62" exitCode=0 Nov 03 01:18:38.147145 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:18:38.147121 5437 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-16.us-west-2.compute.internal\" not found" Nov 03 01:18:38.149155 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.149139 5437 manager.go:324] Recovery completed Nov 03 01:18:38.169979 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.169921 5437 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 01:18:38.172492 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.172473 5437 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-16.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 01:18:38.172576 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.172507 5437 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-16.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 01:18:38.172576 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.172521 5437 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-16.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 01:18:38.174816 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.174800 5437 cpu_manager.go:226] "Starting" policy="none" Nov 03 01:18:38.174816 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.174813 5437 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 03 01:18:38.174910 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.174822 5437 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 01:18:38.174910 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.174830 5437 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 03 01:18:38.174974 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.174946 5437 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 03 01:18:38.174974 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.174957 5437 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 03 01:18:38.175031 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.174975 5437 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 03 01:18:38.175031 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.174981 5437 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 03 01:18:38.175031 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.174986 5437 policy_none.go:50] "Start" Nov 03 01:18:38.175031 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.174993 5437 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 03 01:18:38.175031 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.175002 5437 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 03 01:18:38.175157 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.175126 5437 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 03 01:18:38.175157 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.175133 5437 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 03 01:18:38.175157 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.175139 5437 policy_none.go:44] "Start" Nov 03 01:18:38.176042 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.176025 5437 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-16.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 01:18:38.176094 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.176056 5437 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-16.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 01:18:38.176094 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.176068 5437 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-16.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 01:18:38.179534 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.179502 5437 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 03 01:18:38.181386 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:18:38.181355 5437 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Nov 03 01:18:38.182028 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.182013 5437 manager.go:341] "Starting Device Plugin manager" Nov 03 01:18:38.182102 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:18:38.182035 5437 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 03 01:18:38.182102 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.182044 5437 server.go:87] "Starting device plugin registration server" Nov 03 01:18:38.182301 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.182288 5437 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 03 01:18:38.182389 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.182299 5437 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 03 01:18:38.182453 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.182417 5437 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 03 01:18:38.182548 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.182513 5437 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 03 01:18:38.182625 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.182556 5437 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 03 01:18:38.182625 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.182563 5437 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 03 01:18:38.182726 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.182615 5437 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T01:18:38.182545599Z","UUID":"7dc9b110-b54a-4185-8eaf-8ca0b6514994","Handler":null,"Name":"","Endpoint":""} Nov 03 01:18:38.186179 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:18:38.186154 5437 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 03 01:18:38.186281 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:18:38.186199 5437 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-16.us-west-2.compute.internal\" not found" Nov 03 01:18:38.186574 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.186555 5437 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 03 01:18:38.186672 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.186592 5437 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 03 01:18:38.252025 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.251986 5437 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 01:18:38.252025 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.252017 5437 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 01:18:38.252735 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.252712 5437 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 03 01:18:38.282456 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.282430 5437 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 01:18:38.284530 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.284513 5437 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-16.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 01:18:38.284587 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.284548 5437 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-16.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 01:18:38.284587 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.284562 5437 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-16.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 01:18:38.284643 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.284601 5437 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-136-16.us-west-2.compute.internal" Nov 03 01:18:38.302196 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.302175 5437 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 01:18:38.355204 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.355168 5437 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 01:18:38.382313 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.382255 5437 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-16.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-16.us-west-2.compute.internal"] Nov 03 01:18:38.382932 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.382886 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-16.us-west-2.compute.internal" event={"ID":"7f364a286e3c240d587086e0ec2180aa","Type":"ContainerStarted","Data":"b235ef66737c2cad06a4a918d29bae2112b4568db459177d9e5e853b83cf8792"} Nov 03 01:18:38.382996 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.382937 5437 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-16.us-west-2.compute.internal" Nov 03 01:18:38.382996 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.382949 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-16.us-west-2.compute.internal" event={"ID":"7f364a286e3c240d587086e0ec2180aa","Type":"ContainerStarted","Data":"640208824d01bacb81bae87037315951d0d6301e18043589626cbb13a7a78d2b"} Nov 03 01:18:38.382996 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.382972 5437 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-16.us-west-2.compute.internal" Nov 03 01:18:38.383102 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.383005 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-16.us-west-2.compute.internal" event={"ID":"f7e070a0c923c611d3cf96ab59dc8b57","Type":"ContainerStarted","Data":"3bad95a8bbb41660529b2d2ea4631a6b90124d9ab2f6ad4cef3327f64d303298"} Nov 03 01:18:38.383102 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.383020 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-16.us-west-2.compute.internal" event={"ID":"f7e070a0c923c611d3cf96ab59dc8b57","Type":"ContainerDied","Data":"6fb6612237ca18debdc4ee17b23d29d97c5806aea283b7663f3d2636bc452fbf"} Nov 03 01:18:38.383102 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.383032 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-16.us-west-2.compute.internal" event={"ID":"f7e070a0c923c611d3cf96ab59dc8b57","Type":"ContainerStarted","Data":"a57cfab1072e0e4b06e4b4217747174d1f2639b037b9403811af934a57dc23e5"} Nov 03 01:18:38.408857 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.408830 5437 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-136-16.us-west-2.compute.internal" Nov 03 01:18:38.408958 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.408905 5437 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-136-16.us-west-2.compute.internal" Nov 03 01:18:38.409555 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.409526 5437 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-16.us-west-2.compute.internal" event="NodeNotReady" Nov 03 01:18:38.409647 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.409564 5437 setters.go:546] "Node became not ready" node="ip-10-0-136-16.us-west-2.compute.internal" condition={"type":"Ready","status":"False","lastHeartbeatTime":"2026-11-03T01:18:38Z","lastTransitionTime":"2026-11-03T01:18:38Z","reason":"KubeletNotReady","message":"CSINode is not yet initialized"} Nov 03 01:18:38.445621 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.445548 5437 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 01:18:38.445621 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:18:38.445606 5437 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-136-16.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-16.us-west-2.compute.internal" Nov 03 01:18:38.445980 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.445960 5437 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 01:18:38.446030 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:18:38.446003 5437 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-136-16.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-16.us-west-2.compute.internal" Nov 03 01:18:38.454260 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.454240 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7f364a286e3c240d587086e0ec2180aa-config\") pod \"kube-apiserver-proxy-ip-10-0-136-16.us-west-2.compute.internal\" (UID: \"7f364a286e3c240d587086e0ec2180aa\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-16.us-west-2.compute.internal" Nov 03 01:18:38.454321 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.454266 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f7e070a0c923c611d3cf96ab59dc8b57-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-16.us-west-2.compute.internal\" (UID: \"f7e070a0c923c611d3cf96ab59dc8b57\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-16.us-west-2.compute.internal" Nov 03 01:18:38.454321 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.454284 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f7e070a0c923c611d3cf96ab59dc8b57-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-16.us-west-2.compute.internal\" (UID: \"f7e070a0c923c611d3cf96ab59dc8b57\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-16.us-west-2.compute.internal" Nov 03 01:18:38.472488 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.472458 5437 nodeinfomanager.go:471] existing CSINode "ip-10-0-136-16.us-west-2.compute.internal" is owned by different node (oldNodeID="819b34e9-f465-454f-99fb-693a800d01f2", newNodeID=""), cleaning up... Nov 03 01:18:38.555435 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.555336 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7f364a286e3c240d587086e0ec2180aa-config\") pod \"kube-apiserver-proxy-ip-10-0-136-16.us-west-2.compute.internal\" (UID: \"7f364a286e3c240d587086e0ec2180aa\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-16.us-west-2.compute.internal" Nov 03 01:18:38.555626 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.555465 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f7e070a0c923c611d3cf96ab59dc8b57-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-16.us-west-2.compute.internal\" (UID: \"f7e070a0c923c611d3cf96ab59dc8b57\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-16.us-west-2.compute.internal" Nov 03 01:18:38.555626 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.555408 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7f364a286e3c240d587086e0ec2180aa-config\") pod \"kube-apiserver-proxy-ip-10-0-136-16.us-west-2.compute.internal\" (UID: \"7f364a286e3c240d587086e0ec2180aa\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-16.us-west-2.compute.internal" Nov 03 01:18:38.555626 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.555494 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f7e070a0c923c611d3cf96ab59dc8b57-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-16.us-west-2.compute.internal\" (UID: \"f7e070a0c923c611d3cf96ab59dc8b57\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-16.us-west-2.compute.internal" Nov 03 01:18:38.555626 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.555537 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f7e070a0c923c611d3cf96ab59dc8b57-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-16.us-west-2.compute.internal\" (UID: \"f7e070a0c923c611d3cf96ab59dc8b57\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-16.us-west-2.compute.internal" Nov 03 01:18:38.555626 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:38.555542 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f7e070a0c923c611d3cf96ab59dc8b57-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-16.us-west-2.compute.internal\" (UID: \"f7e070a0c923c611d3cf96ab59dc8b57\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-16.us-west-2.compute.internal" Nov 03 01:18:39.031692 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.031661 5437 apiserver.go:52] "Watching apiserver" Nov 03 01:18:39.139328 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.139294 5437 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 03 01:18:39.139958 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.139936 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-136-16.us-west-2.compute.internal","openshift-monitoring/kube-state-metrics-5dc8d7d89b-htfx6","kube-system/global-pull-secret-syncer-dfr9t","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2jrfl","openshift-console/console-69fcb6c6b8-jqgxm","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-16.us-west-2.compute.internal","openshift-image-registry/node-ca-jpcqz","openshift-multus/network-metrics-daemon-8g56n","openshift-insights/insights-runtime-extractor-bbfdw","openshift-monitoring/monitoring-plugin-77f4f76594-ssqxs","kube-system/konnectivity-agent-dd9jt","openshift-monitoring/node-exporter-5z5ct","openshift-dns/dns-default-kmpbz","openshift-network-operator/iptables-alerter-x4tvh","openshift-multus/multus-44p7d","openshift-dns/node-resolver-5qr9g","openshift-ingress-canary/ingress-canary-4vr4n","openshift-cluster-node-tuning-operator/tuned-c6r2x","openshift-monitoring/openshift-state-metrics-7bd7489957-lcwkf","openshift-monitoring/thanos-querier-8bbf59f7f-cmcdf","openshift-multus/multus-additional-cni-plugins-jnvg6","openshift-network-diagnostics/network-check-target-kg99x","openshift-ovn-kubernetes/ovnkube-node-n2cjg"] Nov 03 01:18:39.140844 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.140824 5437 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-16.us-west-2.compute.internal" Nov 03 01:18:39.140992 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.140856 5437 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-16.us-west-2.compute.internal" Nov 03 01:18:39.158484 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.158455 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6179266f-07ff-48cd-bf03-3f1e961529d7-original-pull-secret\") pod \"global-pull-secret-syncer-dfr9t\" (UID: \"6179266f-07ff-48cd-bf03-3f1e961529d7\") " pod="kube-system/global-pull-secret-syncer-dfr9t" Nov 03 01:18:39.158484 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.158486 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e17d6c62-2952-420c-beda-49db831c48ac-multus-cni-dir\") pod \"multus-44p7d\" (UID: \"e17d6c62-2952-420c-beda-49db831c48ac\") " pod="openshift-multus/multus-44p7d" Nov 03 01:18:39.158722 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.158503 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d157814d-82d6-4af5-a648-c03ad35f3a41-etc-systemd\") pod \"tuned-c6r2x\" (UID: \"d157814d-82d6-4af5-a648-c03ad35f3a41\") " pod="openshift-cluster-node-tuning-operator/tuned-c6r2x" Nov 03 01:18:39.158722 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.158517 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d157814d-82d6-4af5-a648-c03ad35f3a41-run\") pod \"tuned-c6r2x\" (UID: \"d157814d-82d6-4af5-a648-c03ad35f3a41\") " pod="openshift-cluster-node-tuning-operator/tuned-c6r2x" Nov 03 01:18:39.158722 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.158535 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/bff481c2-d05e-4b45-827d-f5c8a0f60d0b-serviceca\") pod \"node-ca-jpcqz\" (UID: \"bff481c2-d05e-4b45-827d-f5c8a0f60d0b\") " pod="openshift-image-registry/node-ca-jpcqz" Nov 03 01:18:39.158722 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.158571 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4c25011b-c85c-4736-9f72-842a96949072-cnibin\") pod \"multus-additional-cni-plugins-jnvg6\" (UID: \"4c25011b-c85c-4736-9f72-842a96949072\") " pod="openshift-multus/multus-additional-cni-plugins-jnvg6" Nov 03 01:18:39.158722 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.158605 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4c25011b-c85c-4736-9f72-842a96949072-cni-binary-copy\") pod \"multus-additional-cni-plugins-jnvg6\" (UID: \"4c25011b-c85c-4736-9f72-842a96949072\") " pod="openshift-multus/multus-additional-cni-plugins-jnvg6" Nov 03 01:18:39.158722 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.158651 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/abdd79d9-8d87-4133-85a2-643d26fc9faa-host-run-netns\") pod \"ovnkube-node-n2cjg\" (UID: \"abdd79d9-8d87-4133-85a2-643d26fc9faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cjg" Nov 03 01:18:39.158722 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.158707 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e17d6c62-2952-420c-beda-49db831c48ac-cnibin\") pod \"multus-44p7d\" (UID: \"e17d6c62-2952-420c-beda-49db831c48ac\") " pod="openshift-multus/multus-44p7d" Nov 03 01:18:39.159022 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.158795 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7skt\" (UniqueName: \"kubernetes.io/projected/e17d6c62-2952-420c-beda-49db831c48ac-kube-api-access-g7skt\") pod \"multus-44p7d\" (UID: \"e17d6c62-2952-420c-beda-49db831c48ac\") " pod="openshift-multus/multus-44p7d" Nov 03 01:18:39.159022 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.158811 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d157814d-82d6-4af5-a648-c03ad35f3a41-tmp\") pod \"tuned-c6r2x\" (UID: \"d157814d-82d6-4af5-a648-c03ad35f3a41\") " pod="openshift-cluster-node-tuning-operator/tuned-c6r2x" Nov 03 01:18:39.159022 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.158826 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/abdd79d9-8d87-4133-85a2-643d26fc9faa-var-lib-openvswitch\") pod \"ovnkube-node-n2cjg\" (UID: \"abdd79d9-8d87-4133-85a2-643d26fc9faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cjg" Nov 03 01:18:39.159022 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.158880 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d157814d-82d6-4af5-a648-c03ad35f3a41-etc-sysctl-conf\") pod \"tuned-c6r2x\" (UID: \"d157814d-82d6-4af5-a648-c03ad35f3a41\") " pod="openshift-cluster-node-tuning-operator/tuned-c6r2x" Nov 03 01:18:39.159022 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.158934 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cct5g\" (UniqueName: \"kubernetes.io/projected/d157814d-82d6-4af5-a648-c03ad35f3a41-kube-api-access-cct5g\") pod \"tuned-c6r2x\" (UID: \"d157814d-82d6-4af5-a648-c03ad35f3a41\") " pod="openshift-cluster-node-tuning-operator/tuned-c6r2x" Nov 03 01:18:39.159022 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.158966 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1febd8fa-5641-449b-b1fd-c7d02603fd1c-registration-dir\") pod \"aws-ebs-csi-driver-node-2jrfl\" (UID: \"1febd8fa-5641-449b-b1fd-c7d02603fd1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2jrfl" Nov 03 01:18:39.159022 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.158988 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bff481c2-d05e-4b45-827d-f5c8a0f60d0b-host\") pod \"node-ca-jpcqz\" (UID: \"bff481c2-d05e-4b45-827d-f5c8a0f60d0b\") " pod="openshift-image-registry/node-ca-jpcqz" Nov 03 01:18:39.159022 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.159019 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4c25011b-c85c-4736-9f72-842a96949072-system-cni-dir\") pod \"multus-additional-cni-plugins-jnvg6\" (UID: \"4c25011b-c85c-4736-9f72-842a96949072\") " pod="openshift-multus/multus-additional-cni-plugins-jnvg6" Nov 03 01:18:39.159453 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.159048 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d157814d-82d6-4af5-a648-c03ad35f3a41-lib-modules\") pod \"tuned-c6r2x\" (UID: \"d157814d-82d6-4af5-a648-c03ad35f3a41\") " pod="openshift-cluster-node-tuning-operator/tuned-c6r2x" Nov 03 01:18:39.159453 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.159071 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/acada406-fef9-4a03-ae21-c2351ef0a4cf-tmp-dir\") pod \"node-resolver-5qr9g\" (UID: \"acada406-fef9-4a03-ae21-c2351ef0a4cf\") " pod="openshift-dns/node-resolver-5qr9g" Nov 03 01:18:39.159453 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.159088 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d157814d-82d6-4af5-a648-c03ad35f3a41-tmp\") pod \"tuned-c6r2x\" (UID: \"d157814d-82d6-4af5-a648-c03ad35f3a41\") " pod="openshift-cluster-node-tuning-operator/tuned-c6r2x" Nov 03 01:18:39.159453 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.159093 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/abdd79d9-8d87-4133-85a2-643d26fc9faa-run-openvswitch\") pod \"ovnkube-node-n2cjg\" (UID: \"abdd79d9-8d87-4133-85a2-643d26fc9faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cjg" Nov 03 01:18:39.159453 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.159136 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/20152fc3-0b5f-4c58-94fb-e02094377c74-iptables-alerter-script\") pod \"iptables-alerter-x4tvh\" (UID: \"20152fc3-0b5f-4c58-94fb-e02094377c74\") " pod="openshift-network-operator/iptables-alerter-x4tvh" Nov 03 01:18:39.159453 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.159147 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/acada406-fef9-4a03-ae21-c2351ef0a4cf-tmp-dir\") pod \"node-resolver-5qr9g\" (UID: \"acada406-fef9-4a03-ae21-c2351ef0a4cf\") " pod="openshift-dns/node-resolver-5qr9g" Nov 03 01:18:39.159453 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.159175 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d157814d-82d6-4af5-a648-c03ad35f3a41-sys\") pod \"tuned-c6r2x\" (UID: \"d157814d-82d6-4af5-a648-c03ad35f3a41\") " pod="openshift-cluster-node-tuning-operator/tuned-c6r2x" Nov 03 01:18:39.159453 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.159203 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1febd8fa-5641-449b-b1fd-c7d02603fd1c-socket-dir\") pod \"aws-ebs-csi-driver-node-2jrfl\" (UID: \"1febd8fa-5641-449b-b1fd-c7d02603fd1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2jrfl" Nov 03 01:18:39.159453 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.159229 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4c25011b-c85c-4736-9f72-842a96949072-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jnvg6\" (UID: \"4c25011b-c85c-4736-9f72-842a96949072\") " pod="openshift-multus/multus-additional-cni-plugins-jnvg6" Nov 03 01:18:39.159453 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.159261 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/abdd79d9-8d87-4133-85a2-643d26fc9faa-host-run-ovn-kubernetes\") pod \"ovnkube-node-n2cjg\" (UID: \"abdd79d9-8d87-4133-85a2-643d26fc9faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cjg" Nov 03 01:18:39.159453 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.159306 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e17d6c62-2952-420c-beda-49db831c48ac-multus-socket-dir-parent\") pod \"multus-44p7d\" (UID: \"e17d6c62-2952-420c-beda-49db831c48ac\") " pod="openshift-multus/multus-44p7d" Nov 03 01:18:39.159453 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.159340 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzr84\" (UniqueName: \"kubernetes.io/projected/4c25011b-c85c-4736-9f72-842a96949072-kube-api-access-vzr84\") pod \"multus-additional-cni-plugins-jnvg6\" (UID: \"4c25011b-c85c-4736-9f72-842a96949072\") " pod="openshift-multus/multus-additional-cni-plugins-jnvg6" Nov 03 01:18:39.159453 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.159383 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/abdd79d9-8d87-4133-85a2-643d26fc9faa-etc-openvswitch\") pod \"ovnkube-node-n2cjg\" (UID: \"abdd79d9-8d87-4133-85a2-643d26fc9faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cjg" Nov 03 01:18:39.159453 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.159408 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/abdd79d9-8d87-4133-85a2-643d26fc9faa-ovn-node-metrics-cert\") pod \"ovnkube-node-n2cjg\" (UID: \"abdd79d9-8d87-4133-85a2-643d26fc9faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cjg" Nov 03 01:18:39.159453 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.159439 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/abdd79d9-8d87-4133-85a2-643d26fc9faa-ovnkube-script-lib\") pod \"ovnkube-node-n2cjg\" (UID: \"abdd79d9-8d87-4133-85a2-643d26fc9faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cjg" Nov 03 01:18:39.160081 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.159503 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/20152fc3-0b5f-4c58-94fb-e02094377c74-host-slash\") pod \"iptables-alerter-x4tvh\" (UID: \"20152fc3-0b5f-4c58-94fb-e02094377c74\") " pod="openshift-network-operator/iptables-alerter-x4tvh" Nov 03 01:18:39.160081 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.159520 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e17d6c62-2952-420c-beda-49db831c48ac-etc-kubernetes\") pod \"multus-44p7d\" (UID: \"e17d6c62-2952-420c-beda-49db831c48ac\") " pod="openshift-multus/multus-44p7d" Nov 03 01:18:39.160081 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.159535 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d157814d-82d6-4af5-a648-c03ad35f3a41-etc-tuned\") pod \"tuned-c6r2x\" (UID: \"d157814d-82d6-4af5-a648-c03ad35f3a41\") " pod="openshift-cluster-node-tuning-operator/tuned-c6r2x" Nov 03 01:18:39.160081 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.159549 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1febd8fa-5641-449b-b1fd-c7d02603fd1c-sys-fs\") pod \"aws-ebs-csi-driver-node-2jrfl\" (UID: \"1febd8fa-5641-449b-b1fd-c7d02603fd1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2jrfl" Nov 03 01:18:39.160081 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.159564 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4c25011b-c85c-4736-9f72-842a96949072-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jnvg6\" (UID: \"4c25011b-c85c-4736-9f72-842a96949072\") " pod="openshift-multus/multus-additional-cni-plugins-jnvg6" Nov 03 01:18:39.160081 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.159634 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/acada406-fef9-4a03-ae21-c2351ef0a4cf-hosts-file\") pod \"node-resolver-5qr9g\" (UID: \"acada406-fef9-4a03-ae21-c2351ef0a4cf\") " pod="openshift-dns/node-resolver-5qr9g" Nov 03 01:18:39.160081 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.159666 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rwtq\" (UniqueName: \"kubernetes.io/projected/acada406-fef9-4a03-ae21-c2351ef0a4cf-kube-api-access-9rwtq\") pod \"node-resolver-5qr9g\" (UID: \"acada406-fef9-4a03-ae21-c2351ef0a4cf\") " pod="openshift-dns/node-resolver-5qr9g" Nov 03 01:18:39.160081 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.159732 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d157814d-82d6-4af5-a648-c03ad35f3a41-host\") pod \"tuned-c6r2x\" (UID: \"d157814d-82d6-4af5-a648-c03ad35f3a41\") " pod="openshift-cluster-node-tuning-operator/tuned-c6r2x" Nov 03 01:18:39.160081 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.159735 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d157814d-82d6-4af5-a648-c03ad35f3a41-etc-tuned\") pod \"tuned-c6r2x\" (UID: \"d157814d-82d6-4af5-a648-c03ad35f3a41\") " pod="openshift-cluster-node-tuning-operator/tuned-c6r2x" Nov 03 01:18:39.160081 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.159760 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ecc2615-b458-41f8-839f-e22c53e7ad0e-metrics-certs\") pod \"network-metrics-daemon-8g56n\" (UID: \"6ecc2615-b458-41f8-839f-e22c53e7ad0e\") " pod="openshift-multus/network-metrics-daemon-8g56n" Nov 03 01:18:39.160081 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.159790 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6179266f-07ff-48cd-bf03-3f1e961529d7-kubelet-config\") pod \"global-pull-secret-syncer-dfr9t\" (UID: \"6179266f-07ff-48cd-bf03-3f1e961529d7\") " pod="kube-system/global-pull-secret-syncer-dfr9t" Nov 03 01:18:39.160081 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.159825 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9j8g\" (UniqueName: \"kubernetes.io/projected/abdd79d9-8d87-4133-85a2-643d26fc9faa-kube-api-access-h9j8g\") pod \"ovnkube-node-n2cjg\" (UID: \"abdd79d9-8d87-4133-85a2-643d26fc9faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cjg" Nov 03 01:18:39.160081 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.159846 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e17d6c62-2952-420c-beda-49db831c48ac-host-run-multus-certs\") pod \"multus-44p7d\" (UID: \"e17d6c62-2952-420c-beda-49db831c48ac\") " pod="openshift-multus/multus-44p7d" Nov 03 01:18:39.160081 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.159886 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgd8w\" (UniqueName: \"kubernetes.io/projected/3b2bf2f6-28aa-4480-9f4a-f4e7f659e966-kube-api-access-vgd8w\") pod \"network-check-target-kg99x\" (UID: \"3b2bf2f6-28aa-4480-9f4a-f4e7f659e966\") " pod="openshift-network-diagnostics/network-check-target-kg99x" Nov 03 01:18:39.160081 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.159917 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/abdd79d9-8d87-4133-85a2-643d26fc9faa-host-cni-netd\") pod \"ovnkube-node-n2cjg\" (UID: \"abdd79d9-8d87-4133-85a2-643d26fc9faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cjg" Nov 03 01:18:39.160081 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.159943 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/abdd79d9-8d87-4133-85a2-643d26fc9faa-ovnkube-config\") pod \"ovnkube-node-n2cjg\" (UID: \"abdd79d9-8d87-4133-85a2-643d26fc9faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cjg" Nov 03 01:18:39.160081 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.159967 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/abdd79d9-8d87-4133-85a2-643d26fc9faa-run-systemd\") pod \"ovnkube-node-n2cjg\" (UID: \"abdd79d9-8d87-4133-85a2-643d26fc9faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cjg" Nov 03 01:18:39.160081 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.159990 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/abdd79d9-8d87-4133-85a2-643d26fc9faa-host-slash\") pod \"ovnkube-node-n2cjg\" (UID: \"abdd79d9-8d87-4133-85a2-643d26fc9faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cjg" Nov 03 01:18:39.160081 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.160027 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/abdd79d9-8d87-4133-85a2-643d26fc9faa-run-ovn\") pod \"ovnkube-node-n2cjg\" (UID: \"abdd79d9-8d87-4133-85a2-643d26fc9faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cjg" Nov 03 01:18:39.160081 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.160085 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e17d6c62-2952-420c-beda-49db831c48ac-cni-binary-copy\") pod \"multus-44p7d\" (UID: \"e17d6c62-2952-420c-beda-49db831c48ac\") " pod="openshift-multus/multus-44p7d" Nov 03 01:18:39.160913 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.160134 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e17d6c62-2952-420c-beda-49db831c48ac-host-run-k8s-cni-cncf-io\") pod \"multus-44p7d\" (UID: \"e17d6c62-2952-420c-beda-49db831c48ac\") " pod="openshift-multus/multus-44p7d" Nov 03 01:18:39.160913 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.160158 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e17d6c62-2952-420c-beda-49db831c48ac-multus-daemon-config\") pod \"multus-44p7d\" (UID: \"e17d6c62-2952-420c-beda-49db831c48ac\") " pod="openshift-multus/multus-44p7d" Nov 03 01:18:39.160913 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.160200 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/abdd79d9-8d87-4133-85a2-643d26fc9faa-host-kubelet\") pod \"ovnkube-node-n2cjg\" (UID: \"abdd79d9-8d87-4133-85a2-643d26fc9faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cjg" Nov 03 01:18:39.160913 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.160218 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/abdd79d9-8d87-4133-85a2-643d26fc9faa-env-overrides\") pod \"ovnkube-node-n2cjg\" (UID: \"abdd79d9-8d87-4133-85a2-643d26fc9faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cjg" Nov 03 01:18:39.160913 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.160233 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2r8s\" (UniqueName: \"kubernetes.io/projected/20152fc3-0b5f-4c58-94fb-e02094377c74-kube-api-access-m2r8s\") pod \"iptables-alerter-x4tvh\" (UID: \"20152fc3-0b5f-4c58-94fb-e02094377c74\") " pod="openshift-network-operator/iptables-alerter-x4tvh" Nov 03 01:18:39.160913 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.160249 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e17d6c62-2952-420c-beda-49db831c48ac-host-var-lib-cni-multus\") pod \"multus-44p7d\" (UID: \"e17d6c62-2952-420c-beda-49db831c48ac\") " pod="openshift-multus/multus-44p7d" Nov 03 01:18:39.160913 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.160263 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d157814d-82d6-4af5-a648-c03ad35f3a41-etc-sysconfig\") pod \"tuned-c6r2x\" (UID: \"d157814d-82d6-4af5-a648-c03ad35f3a41\") " pod="openshift-cluster-node-tuning-operator/tuned-c6r2x" Nov 03 01:18:39.160913 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.160344 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d157814d-82d6-4af5-a648-c03ad35f3a41-etc-sysctl-d\") pod \"tuned-c6r2x\" (UID: \"d157814d-82d6-4af5-a648-c03ad35f3a41\") " pod="openshift-cluster-node-tuning-operator/tuned-c6r2x" Nov 03 01:18:39.160913 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.160402 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d157814d-82d6-4af5-a648-c03ad35f3a41-etc-kubernetes\") pod \"tuned-c6r2x\" (UID: \"d157814d-82d6-4af5-a648-c03ad35f3a41\") " pod="openshift-cluster-node-tuning-operator/tuned-c6r2x" Nov 03 01:18:39.160913 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.160433 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d157814d-82d6-4af5-a648-c03ad35f3a41-var-lib-kubelet\") pod \"tuned-c6r2x\" (UID: \"d157814d-82d6-4af5-a648-c03ad35f3a41\") " pod="openshift-cluster-node-tuning-operator/tuned-c6r2x" Nov 03 01:18:39.160913 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.160448 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/abdd79d9-8d87-4133-85a2-643d26fc9faa-node-log\") pod \"ovnkube-node-n2cjg\" (UID: \"abdd79d9-8d87-4133-85a2-643d26fc9faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cjg" Nov 03 01:18:39.160913 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.160477 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e17d6c62-2952-420c-beda-49db831c48ac-hostroot\") pod \"multus-44p7d\" (UID: \"e17d6c62-2952-420c-beda-49db831c48ac\") " pod="openshift-multus/multus-44p7d" Nov 03 01:18:39.160913 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.160497 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e17d6c62-2952-420c-beda-49db831c48ac-multus-conf-dir\") pod \"multus-44p7d\" (UID: \"e17d6c62-2952-420c-beda-49db831c48ac\") " pod="openshift-multus/multus-44p7d" Nov 03 01:18:39.160913 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.160516 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1febd8fa-5641-449b-b1fd-c7d02603fd1c-device-dir\") pod \"aws-ebs-csi-driver-node-2jrfl\" (UID: \"1febd8fa-5641-449b-b1fd-c7d02603fd1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2jrfl" Nov 03 01:18:39.160913 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.160569 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdm2b\" (UniqueName: \"kubernetes.io/projected/1febd8fa-5641-449b-b1fd-c7d02603fd1c-kube-api-access-bdm2b\") pod \"aws-ebs-csi-driver-node-2jrfl\" (UID: \"1febd8fa-5641-449b-b1fd-c7d02603fd1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2jrfl" Nov 03 01:18:39.160913 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.160607 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88c5l\" (UniqueName: \"kubernetes.io/projected/bff481c2-d05e-4b45-827d-f5c8a0f60d0b-kube-api-access-88c5l\") pod \"node-ca-jpcqz\" (UID: \"bff481c2-d05e-4b45-827d-f5c8a0f60d0b\") " pod="openshift-image-registry/node-ca-jpcqz" Nov 03 01:18:39.160913 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.160682 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b56kh\" (UniqueName: \"kubernetes.io/projected/6ecc2615-b458-41f8-839f-e22c53e7ad0e-kube-api-access-b56kh\") pod \"network-metrics-daemon-8g56n\" (UID: \"6ecc2615-b458-41f8-839f-e22c53e7ad0e\") " pod="openshift-multus/network-metrics-daemon-8g56n" Nov 03 01:18:39.160913 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.160722 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e17d6c62-2952-420c-beda-49db831c48ac-system-cni-dir\") pod \"multus-44p7d\" (UID: \"e17d6c62-2952-420c-beda-49db831c48ac\") " pod="openshift-multus/multus-44p7d" Nov 03 01:18:39.160913 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.160768 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e17d6c62-2952-420c-beda-49db831c48ac-host-run-netns\") pod \"multus-44p7d\" (UID: \"e17d6c62-2952-420c-beda-49db831c48ac\") " pod="openshift-multus/multus-44p7d" Nov 03 01:18:39.160913 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.160809 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1febd8fa-5641-449b-b1fd-c7d02603fd1c-etc-selinux\") pod \"aws-ebs-csi-driver-node-2jrfl\" (UID: \"1febd8fa-5641-449b-b1fd-c7d02603fd1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2jrfl" Nov 03 01:18:39.160913 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.160841 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/abdd79d9-8d87-4133-85a2-643d26fc9faa-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-n2cjg\" (UID: \"abdd79d9-8d87-4133-85a2-643d26fc9faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cjg" Nov 03 01:18:39.160913 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.160872 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6179266f-07ff-48cd-bf03-3f1e961529d7-dbus\") pod \"global-pull-secret-syncer-dfr9t\" (UID: \"6179266f-07ff-48cd-bf03-3f1e961529d7\") " pod="kube-system/global-pull-secret-syncer-dfr9t" Nov 03 01:18:39.160913 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.160898 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/abdd79d9-8d87-4133-85a2-643d26fc9faa-log-socket\") pod \"ovnkube-node-n2cjg\" (UID: \"abdd79d9-8d87-4133-85a2-643d26fc9faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cjg" Nov 03 01:18:39.161636 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.160933 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/abdd79d9-8d87-4133-85a2-643d26fc9faa-host-cni-bin\") pod \"ovnkube-node-n2cjg\" (UID: \"abdd79d9-8d87-4133-85a2-643d26fc9faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cjg" Nov 03 01:18:39.161636 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.160970 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4c25011b-c85c-4736-9f72-842a96949072-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jnvg6\" (UID: \"4c25011b-c85c-4736-9f72-842a96949072\") " pod="openshift-multus/multus-additional-cni-plugins-jnvg6" Nov 03 01:18:39.161636 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.161005 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e17d6c62-2952-420c-beda-49db831c48ac-host-var-lib-cni-bin\") pod \"multus-44p7d\" (UID: \"e17d6c62-2952-420c-beda-49db831c48ac\") " pod="openshift-multus/multus-44p7d" Nov 03 01:18:39.161636 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.161042 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e17d6c62-2952-420c-beda-49db831c48ac-host-var-lib-kubelet\") pod \"multus-44p7d\" (UID: \"e17d6c62-2952-420c-beda-49db831c48ac\") " pod="openshift-multus/multus-44p7d" Nov 03 01:18:39.161636 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.161062 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/abdd79d9-8d87-4133-85a2-643d26fc9faa-systemd-units\") pod \"ovnkube-node-n2cjg\" (UID: \"abdd79d9-8d87-4133-85a2-643d26fc9faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cjg" Nov 03 01:18:39.161636 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.161076 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d157814d-82d6-4af5-a648-c03ad35f3a41-etc-modprobe-d\") pod \"tuned-c6r2x\" (UID: \"d157814d-82d6-4af5-a648-c03ad35f3a41\") " pod="openshift-cluster-node-tuning-operator/tuned-c6r2x" Nov 03 01:18:39.161636 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.161091 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1febd8fa-5641-449b-b1fd-c7d02603fd1c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2jrfl\" (UID: \"1febd8fa-5641-449b-b1fd-c7d02603fd1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2jrfl" Nov 03 01:18:39.195659 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.195631 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 03 01:18:39.195659 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.195654 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 01:18:39.195931 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.195670 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 03 01:18:39.195931 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.195678 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 01:18:39.195931 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.195693 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 03 01:18:39.195931 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.195755 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 03 01:18:39.195931 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.195770 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-fqzf6\"" Nov 03 01:18:39.195931 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.195773 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-mmkcs\"" Nov 03 01:18:39.195931 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.195799 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-r4kvh\"" Nov 03 01:18:39.195931 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.195814 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 01:18:39.195931 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.195827 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-t8fl2\"" Nov 03 01:18:39.196194 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.196103 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 03 01:18:39.199409 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.199393 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6179266f-07ff-48cd-bf03-3f1e961529d7-original-pull-secret\") pod \"global-pull-secret-syncer-dfr9t\" (UID: \"6179266f-07ff-48cd-bf03-3f1e961529d7\") " pod="kube-system/global-pull-secret-syncer-dfr9t" Nov 03 01:18:39.200088 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.200074 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 03 01:18:39.203879 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.203861 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-bdhgr\"" Nov 03 01:18:39.204176 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.204160 5437 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 01:18:39.204241 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.204175 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 03 01:18:39.204314 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:18:39.204300 5437 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-136-16.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-16.us-west-2.compute.internal" Nov 03 01:18:39.204375 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.204302 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-c92gz\"" Nov 03 01:18:39.204469 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.204452 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-th4bh\"" Nov 03 01:18:39.204511 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.204489 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-7sd2v\"" Nov 03 01:18:39.204541 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.204509 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-2fc25\"" Nov 03 01:18:39.204541 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.204538 5437 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 01:18:39.204630 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.204560 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-nkj8v\"" Nov 03 01:18:39.204630 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.204610 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-9tz7d\"" Nov 03 01:18:39.204723 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.204630 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 03 01:18:39.204723 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:18:39.204692 5437 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-136-16.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-16.us-west-2.compute.internal" Nov 03 01:18:39.206866 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.206823 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 03 01:18:39.206866 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.206886 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 03 01:18:39.207135 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.206915 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 03 01:18:39.207135 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.206923 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 03 01:18:39.207135 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.207110 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 03 01:18:39.207135 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.207118 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 03 01:18:39.207353 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.207162 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 03 01:18:39.207353 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.207243 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 03 01:18:39.207353 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.207277 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 03 01:18:39.207353 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.207314 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 03 01:18:39.207353 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.207328 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 03 01:18:39.207353 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.207319 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 03 01:18:39.207666 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.207389 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 03 01:18:39.207666 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.207397 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 03 01:18:39.207666 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.207403 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 03 01:18:39.207666 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.207644 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 03 01:18:39.207666 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.207660 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 03 01:18:39.209753 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.209723 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/abdd79d9-8d87-4133-85a2-643d26fc9faa-ovnkube-script-lib\") pod \"ovnkube-node-n2cjg\" (UID: \"abdd79d9-8d87-4133-85a2-643d26fc9faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cjg" Nov 03 01:18:39.209861 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.209798 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/20152fc3-0b5f-4c58-94fb-e02094377c74-iptables-alerter-script\") pod \"iptables-alerter-x4tvh\" (UID: \"20152fc3-0b5f-4c58-94fb-e02094377c74\") " pod="openshift-network-operator/iptables-alerter-x4tvh" Nov 03 01:18:39.209861 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.209799 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4c25011b-c85c-4736-9f72-842a96949072-cni-binary-copy\") pod \"multus-additional-cni-plugins-jnvg6\" (UID: \"4c25011b-c85c-4736-9f72-842a96949072\") " pod="openshift-multus/multus-additional-cni-plugins-jnvg6" Nov 03 01:18:39.209980 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.209861 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/abdd79d9-8d87-4133-85a2-643d26fc9faa-ovn-node-metrics-cert\") pod \"ovnkube-node-n2cjg\" (UID: \"abdd79d9-8d87-4133-85a2-643d26fc9faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cjg" Nov 03 01:18:39.209980 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.209864 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4c25011b-c85c-4736-9f72-842a96949072-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jnvg6\" (UID: \"4c25011b-c85c-4736-9f72-842a96949072\") " pod="openshift-multus/multus-additional-cni-plugins-jnvg6" Nov 03 01:18:39.209980 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.209934 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4c25011b-c85c-4736-9f72-842a96949072-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jnvg6\" (UID: \"4c25011b-c85c-4736-9f72-842a96949072\") " pod="openshift-multus/multus-additional-cni-plugins-jnvg6" Nov 03 01:18:39.210115 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.210093 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/bff481c2-d05e-4b45-827d-f5c8a0f60d0b-serviceca\") pod \"node-ca-jpcqz\" (UID: \"bff481c2-d05e-4b45-827d-f5c8a0f60d0b\") " pod="openshift-image-registry/node-ca-jpcqz" Nov 03 01:18:39.211106 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.211068 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/abdd79d9-8d87-4133-85a2-643d26fc9faa-env-overrides\") pod \"ovnkube-node-n2cjg\" (UID: \"abdd79d9-8d87-4133-85a2-643d26fc9faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cjg" Nov 03 01:18:39.211106 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.211100 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/abdd79d9-8d87-4133-85a2-643d26fc9faa-ovnkube-config\") pod \"ovnkube-node-n2cjg\" (UID: \"abdd79d9-8d87-4133-85a2-643d26fc9faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cjg" Nov 03 01:18:39.211217 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.211109 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e17d6c62-2952-420c-beda-49db831c48ac-cni-binary-copy\") pod \"multus-44p7d\" (UID: \"e17d6c62-2952-420c-beda-49db831c48ac\") " pod="openshift-multus/multus-44p7d" Nov 03 01:18:39.211217 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.211130 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e17d6c62-2952-420c-beda-49db831c48ac-multus-daemon-config\") pod \"multus-44p7d\" (UID: \"e17d6c62-2952-420c-beda-49db831c48ac\") " pod="openshift-multus/multus-44p7d" Nov 03 01:18:39.245038 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.245003 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 03 01:18:39.245311 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.245042 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 03 01:18:39.245311 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.245185 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 03 01:18:39.245311 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.245286 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-tcxvw\"" Nov 03 01:18:39.245515 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.245327 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 03 01:18:39.245515 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.245345 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Nov 03 01:18:39.245515 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.245383 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 03 01:18:39.245515 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.245388 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 03 01:18:39.245515 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.245434 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 03 01:18:39.245515 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.245448 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 03 01:18:39.245855 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.245836 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 03 01:18:39.245960 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.245912 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Nov 03 01:18:39.245960 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.245931 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-rgvh9\"" Nov 03 01:18:39.246054 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.245961 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-4pcs8\"" Nov 03 01:18:39.246108 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.246070 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Nov 03 01:18:39.246183 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.246170 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Nov 03 01:18:39.246222 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.246198 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-w97sx\"" Nov 03 01:18:39.246273 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.246262 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 03 01:18:39.246745 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.246713 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Nov 03 01:18:39.246745 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.246771 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Nov 03 01:18:39.246969 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.246823 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 03 01:18:39.246969 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.246845 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Nov 03 01:18:39.246969 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.246909 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 03 01:18:39.246969 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.246960 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 03 01:18:39.247163 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.247019 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Nov 03 01:18:39.247256 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.247243 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 03 01:18:39.247674 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.247658 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Nov 03 01:18:39.247813 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.247795 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Nov 03 01:18:39.248191 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.248176 5437 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 03 01:18:39.248609 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.248538 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 03 01:18:39.248609 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.248571 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Nov 03 01:18:39.248609 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.248597 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 03 01:18:39.248774 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.248624 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-8hlmd393unt3b\"" Nov 03 01:18:39.248774 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.248538 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-bh2h6\"" Nov 03 01:18:39.248874 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.248799 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Nov 03 01:18:39.248874 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.248865 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-r69hg\"" Nov 03 01:18:39.248968 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.248867 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Nov 03 01:18:39.249020 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.248966 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-9f8wp\"" Nov 03 01:18:39.249020 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.248990 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 03 01:18:39.249121 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.249042 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Nov 03 01:18:39.249121 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.249098 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Nov 03 01:18:39.249191 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.249165 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qwk4g\"" Nov 03 01:18:39.251951 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.251935 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-w7g4c\"" Nov 03 01:18:39.252006 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.251937 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Nov 03 01:18:39.252006 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.251978 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Nov 03 01:18:39.252006 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.251980 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Nov 03 01:18:39.257721 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.257691 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rwtq\" (UniqueName: \"kubernetes.io/projected/acada406-fef9-4a03-ae21-c2351ef0a4cf-kube-api-access-9rwtq\") pod \"node-resolver-5qr9g\" (UID: \"acada406-fef9-4a03-ae21-c2351ef0a4cf\") " pod="openshift-dns/node-resolver-5qr9g" Nov 03 01:18:39.258238 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.258213 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Nov 03 01:18:39.258340 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.258318 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2r8s\" (UniqueName: \"kubernetes.io/projected/20152fc3-0b5f-4c58-94fb-e02094377c74-kube-api-access-m2r8s\") pod \"iptables-alerter-x4tvh\" (UID: \"20152fc3-0b5f-4c58-94fb-e02094377c74\") " pod="openshift-network-operator/iptables-alerter-x4tvh" Nov 03 01:18:39.258427 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.258326 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-fc5zz\"" Nov 03 01:18:39.258427 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.258396 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 01:18:39.259086 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.259060 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b56kh\" (UniqueName: \"kubernetes.io/projected/6ecc2615-b458-41f8-839f-e22c53e7ad0e-kube-api-access-b56kh\") pod \"network-metrics-daemon-8g56n\" (UID: \"6ecc2615-b458-41f8-839f-e22c53e7ad0e\") " pod="openshift-multus/network-metrics-daemon-8g56n" Nov 03 01:18:39.259239 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.259218 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgd8w\" (UniqueName: \"kubernetes.io/projected/3b2bf2f6-28aa-4480-9f4a-f4e7f659e966-kube-api-access-vgd8w\") pod \"network-check-target-kg99x\" (UID: \"3b2bf2f6-28aa-4480-9f4a-f4e7f659e966\") " pod="openshift-network-diagnostics/network-check-target-kg99x" Nov 03 01:18:39.259815 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.259791 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdm2b\" (UniqueName: \"kubernetes.io/projected/1febd8fa-5641-449b-b1fd-c7d02603fd1c-kube-api-access-bdm2b\") pod \"aws-ebs-csi-driver-node-2jrfl\" (UID: \"1febd8fa-5641-449b-b1fd-c7d02603fd1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2jrfl" Nov 03 01:18:39.260061 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.260042 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ecc2615-b458-41f8-839f-e22c53e7ad0e-metrics-certs\") pod \"network-metrics-daemon-8g56n\" (UID: \"6ecc2615-b458-41f8-839f-e22c53e7ad0e\") " pod="openshift-multus/network-metrics-daemon-8g56n" Nov 03 01:18:39.260656 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.260632 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88c5l\" (UniqueName: \"kubernetes.io/projected/bff481c2-d05e-4b45-827d-f5c8a0f60d0b-kube-api-access-88c5l\") pod \"node-ca-jpcqz\" (UID: \"bff481c2-d05e-4b45-827d-f5c8a0f60d0b\") " pod="openshift-image-registry/node-ca-jpcqz" Nov 03 01:18:39.260772 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.260748 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7skt\" (UniqueName: \"kubernetes.io/projected/e17d6c62-2952-420c-beda-49db831c48ac-kube-api-access-g7skt\") pod \"multus-44p7d\" (UID: \"e17d6c62-2952-420c-beda-49db831c48ac\") " pod="openshift-multus/multus-44p7d" Nov 03 01:18:39.261034 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.261010 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cct5g\" (UniqueName: \"kubernetes.io/projected/d157814d-82d6-4af5-a648-c03ad35f3a41-kube-api-access-cct5g\") pod \"tuned-c6r2x\" (UID: \"d157814d-82d6-4af5-a648-c03ad35f3a41\") " pod="openshift-cluster-node-tuning-operator/tuned-c6r2x" Nov 03 01:18:39.261559 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.261540 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d157814d-82d6-4af5-a648-c03ad35f3a41-host\") pod \"tuned-c6r2x\" (UID: \"d157814d-82d6-4af5-a648-c03ad35f3a41\") " pod="openshift-cluster-node-tuning-operator/tuned-c6r2x" Nov 03 01:18:39.261609 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.261569 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6179266f-07ff-48cd-bf03-3f1e961529d7-kubelet-config\") pod \"global-pull-secret-syncer-dfr9t\" (UID: \"6179266f-07ff-48cd-bf03-3f1e961529d7\") " pod="kube-system/global-pull-secret-syncer-dfr9t" Nov 03 01:18:39.261609 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.261587 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e17d6c62-2952-420c-beda-49db831c48ac-host-run-multus-certs\") pod \"multus-44p7d\" (UID: \"e17d6c62-2952-420c-beda-49db831c48ac\") " pod="openshift-multus/multus-44p7d" Nov 03 01:18:39.261711 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.261604 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/766a044f-fc66-4998-8006-343c0329860d-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-htfx6\" (UID: \"766a044f-fc66-4998-8006-343c0329860d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-htfx6" Nov 03 01:18:39.261711 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.261615 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzr84\" (UniqueName: \"kubernetes.io/projected/4c25011b-c85c-4736-9f72-842a96949072-kube-api-access-vzr84\") pod \"multus-additional-cni-plugins-jnvg6\" (UID: \"4c25011b-c85c-4736-9f72-842a96949072\") " pod="openshift-multus/multus-additional-cni-plugins-jnvg6" Nov 03 01:18:39.261711 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.261630 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/8f478179-5984-48dd-b872-8f552954ecea-secret-thanos-querier-tls\") pod \"thanos-querier-8bbf59f7f-cmcdf\" (UID: \"8f478179-5984-48dd-b872-8f552954ecea\") " pod="openshift-monitoring/thanos-querier-8bbf59f7f-cmcdf" Nov 03 01:18:39.261711 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.261655 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/abdd79d9-8d87-4133-85a2-643d26fc9faa-run-systemd\") pod \"ovnkube-node-n2cjg\" (UID: \"abdd79d9-8d87-4133-85a2-643d26fc9faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cjg" Nov 03 01:18:39.261711 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.261664 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d157814d-82d6-4af5-a648-c03ad35f3a41-host\") pod \"tuned-c6r2x\" (UID: \"d157814d-82d6-4af5-a648-c03ad35f3a41\") " pod="openshift-cluster-node-tuning-operator/tuned-c6r2x" Nov 03 01:18:39.261711 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.261677 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/abdd79d9-8d87-4133-85a2-643d26fc9faa-host-slash\") pod \"ovnkube-node-n2cjg\" (UID: \"abdd79d9-8d87-4133-85a2-643d26fc9faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cjg" Nov 03 01:18:39.261711 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.261673 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6179266f-07ff-48cd-bf03-3f1e961529d7-kubelet-config\") pod \"global-pull-secret-syncer-dfr9t\" (UID: \"6179266f-07ff-48cd-bf03-3f1e961529d7\") " pod="kube-system/global-pull-secret-syncer-dfr9t" Nov 03 01:18:39.261711 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.261678 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e17d6c62-2952-420c-beda-49db831c48ac-host-run-multus-certs\") pod \"multus-44p7d\" (UID: \"e17d6c62-2952-420c-beda-49db831c48ac\") " pod="openshift-multus/multus-44p7d" Nov 03 01:18:39.261711 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.261692 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9j8g\" (UniqueName: \"kubernetes.io/projected/abdd79d9-8d87-4133-85a2-643d26fc9faa-kube-api-access-h9j8g\") pod \"ovnkube-node-n2cjg\" (UID: \"abdd79d9-8d87-4133-85a2-643d26fc9faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cjg" Nov 03 01:18:39.262151 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.261716 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/abdd79d9-8d87-4133-85a2-643d26fc9faa-run-ovn\") pod \"ovnkube-node-n2cjg\" (UID: \"abdd79d9-8d87-4133-85a2-643d26fc9faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cjg" Nov 03 01:18:39.262151 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.261720 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/766a044f-fc66-4998-8006-343c0329860d-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-htfx6\" (UID: \"766a044f-fc66-4998-8006-343c0329860d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-htfx6" Nov 03 01:18:39.262151 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.261721 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/abdd79d9-8d87-4133-85a2-643d26fc9faa-run-systemd\") pod \"ovnkube-node-n2cjg\" (UID: \"abdd79d9-8d87-4133-85a2-643d26fc9faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cjg" Nov 03 01:18:39.262151 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.261746 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2cz9\" (UniqueName: \"kubernetes.io/projected/2320cce9-0fdb-470a-ae01-e88118ba0419-kube-api-access-b2cz9\") pod \"insights-runtime-extractor-bbfdw\" (UID: \"2320cce9-0fdb-470a-ae01-e88118ba0419\") " pod="openshift-insights/insights-runtime-extractor-bbfdw" Nov 03 01:18:39.262151 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.261751 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/abdd79d9-8d87-4133-85a2-643d26fc9faa-run-ovn\") pod \"ovnkube-node-n2cjg\" (UID: \"abdd79d9-8d87-4133-85a2-643d26fc9faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cjg" Nov 03 01:18:39.262151 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.261739 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/abdd79d9-8d87-4133-85a2-643d26fc9faa-host-slash\") pod \"ovnkube-node-n2cjg\" (UID: \"abdd79d9-8d87-4133-85a2-643d26fc9faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cjg" Nov 03 01:18:39.262151 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.261846 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1fdb6f2d-9138-4f57-af12-c3909dff6e5f-config-volume\") pod \"dns-default-kmpbz\" (UID: \"1fdb6f2d-9138-4f57-af12-c3909dff6e5f\") " pod="openshift-dns/dns-default-kmpbz" Nov 03 01:18:39.262151 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.261875 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e17d6c62-2952-420c-beda-49db831c48ac-host-var-lib-cni-multus\") pod \"multus-44p7d\" (UID: \"e17d6c62-2952-420c-beda-49db831c48ac\") " pod="openshift-multus/multus-44p7d" Nov 03 01:18:39.262151 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.261908 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/8f478179-5984-48dd-b872-8f552954ecea-secret-thanos-querier-tls\") pod \"thanos-querier-8bbf59f7f-cmcdf\" (UID: \"8f478179-5984-48dd-b872-8f552954ecea\") " pod="openshift-monitoring/thanos-querier-8bbf59f7f-cmcdf" Nov 03 01:18:39.262151 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.261912 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d157814d-82d6-4af5-a648-c03ad35f3a41-etc-sysctl-d\") pod \"tuned-c6r2x\" (UID: \"d157814d-82d6-4af5-a648-c03ad35f3a41\") " pod="openshift-cluster-node-tuning-operator/tuned-c6r2x" Nov 03 01:18:39.262151 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.261952 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e17d6c62-2952-420c-beda-49db831c48ac-host-var-lib-cni-multus\") pod \"multus-44p7d\" (UID: \"e17d6c62-2952-420c-beda-49db831c48ac\") " pod="openshift-multus/multus-44p7d" Nov 03 01:18:39.262151 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.261962 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d157814d-82d6-4af5-a648-c03ad35f3a41-var-lib-kubelet\") pod \"tuned-c6r2x\" (UID: \"d157814d-82d6-4af5-a648-c03ad35f3a41\") " pod="openshift-cluster-node-tuning-operator/tuned-c6r2x" Nov 03 01:18:39.262151 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.261979 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/abdd79d9-8d87-4133-85a2-643d26fc9faa-node-log\") pod \"ovnkube-node-n2cjg\" (UID: \"abdd79d9-8d87-4133-85a2-643d26fc9faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cjg" Nov 03 01:18:39.262151 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.262001 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e17d6c62-2952-420c-beda-49db831c48ac-hostroot\") pod \"multus-44p7d\" (UID: \"e17d6c62-2952-420c-beda-49db831c48ac\") " pod="openshift-multus/multus-44p7d" Nov 03 01:18:39.262151 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.262023 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d157814d-82d6-4af5-a648-c03ad35f3a41-etc-sysctl-d\") pod \"tuned-c6r2x\" (UID: \"d157814d-82d6-4af5-a648-c03ad35f3a41\") " pod="openshift-cluster-node-tuning-operator/tuned-c6r2x" Nov 03 01:18:39.262151 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.262025 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e17d6c62-2952-420c-beda-49db831c48ac-multus-conf-dir\") pod \"multus-44p7d\" (UID: \"e17d6c62-2952-420c-beda-49db831c48ac\") " pod="openshift-multus/multus-44p7d" Nov 03 01:18:39.262151 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.262037 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d157814d-82d6-4af5-a648-c03ad35f3a41-var-lib-kubelet\") pod \"tuned-c6r2x\" (UID: \"d157814d-82d6-4af5-a648-c03ad35f3a41\") " pod="openshift-cluster-node-tuning-operator/tuned-c6r2x" Nov 03 01:18:39.262151 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.262061 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1febd8fa-5641-449b-b1fd-c7d02603fd1c-device-dir\") pod \"aws-ebs-csi-driver-node-2jrfl\" (UID: \"1febd8fa-5641-449b-b1fd-c7d02603fd1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2jrfl" Nov 03 01:18:39.262151 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.262065 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/abdd79d9-8d87-4133-85a2-643d26fc9faa-node-log\") pod \"ovnkube-node-n2cjg\" (UID: \"abdd79d9-8d87-4133-85a2-643d26fc9faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cjg" Nov 03 01:18:39.262151 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.262064 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e17d6c62-2952-420c-beda-49db831c48ac-multus-conf-dir\") pod \"multus-44p7d\" (UID: \"e17d6c62-2952-420c-beda-49db831c48ac\") " pod="openshift-multus/multus-44p7d" Nov 03 01:18:39.262151 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.262070 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e17d6c62-2952-420c-beda-49db831c48ac-hostroot\") pod \"multus-44p7d\" (UID: \"e17d6c62-2952-420c-beda-49db831c48ac\") " pod="openshift-multus/multus-44p7d" Nov 03 01:18:39.262151 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.262083 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fe434c5e-450b-4512-93aa-78d3e56e5002-konnectivity-ca\") pod \"konnectivity-agent-dd9jt\" (UID: \"fe434c5e-450b-4512-93aa-78d3e56e5002\") " pod="kube-system/konnectivity-agent-dd9jt" Nov 03 01:18:39.262151 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.262115 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1febd8fa-5641-449b-b1fd-c7d02603fd1c-device-dir\") pod \"aws-ebs-csi-driver-node-2jrfl\" (UID: \"1febd8fa-5641-449b-b1fd-c7d02603fd1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2jrfl" Nov 03 01:18:39.262151 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.262140 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e17d6c62-2952-420c-beda-49db831c48ac-host-run-netns\") pod \"multus-44p7d\" (UID: \"e17d6c62-2952-420c-beda-49db831c48ac\") " pod="openshift-multus/multus-44p7d" Nov 03 01:18:39.262151 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.262159 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1febd8fa-5641-449b-b1fd-c7d02603fd1c-etc-selinux\") pod \"aws-ebs-csi-driver-node-2jrfl\" (UID: \"1febd8fa-5641-449b-b1fd-c7d02603fd1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2jrfl" Nov 03 01:18:39.263183 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.262175 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e17d6c62-2952-420c-beda-49db831c48ac-host-var-lib-kubelet\") pod \"multus-44p7d\" (UID: \"e17d6c62-2952-420c-beda-49db831c48ac\") " pod="openshift-multus/multus-44p7d" Nov 03 01:18:39.263183 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.262076 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1fdb6f2d-9138-4f57-af12-c3909dff6e5f-config-volume\") pod \"dns-default-kmpbz\" (UID: \"1fdb6f2d-9138-4f57-af12-c3909dff6e5f\") " pod="openshift-dns/dns-default-kmpbz" Nov 03 01:18:39.263183 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.262189 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e17d6c62-2952-420c-beda-49db831c48ac-host-run-netns\") pod \"multus-44p7d\" (UID: \"e17d6c62-2952-420c-beda-49db831c48ac\") " pod="openshift-multus/multus-44p7d" Nov 03 01:18:39.263183 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.262194 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2320cce9-0fdb-470a-ae01-e88118ba0419-crio-socket\") pod \"insights-runtime-extractor-bbfdw\" (UID: \"2320cce9-0fdb-470a-ae01-e88118ba0419\") " pod="openshift-insights/insights-runtime-extractor-bbfdw" Nov 03 01:18:39.263183 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.262228 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fe434c5e-450b-4512-93aa-78d3e56e5002-konnectivity-ca\") pod \"konnectivity-agent-dd9jt\" (UID: \"fe434c5e-450b-4512-93aa-78d3e56e5002\") " pod="kube-system/konnectivity-agent-dd9jt" Nov 03 01:18:39.263183 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.262233 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1febd8fa-5641-449b-b1fd-c7d02603fd1c-etc-selinux\") pod \"aws-ebs-csi-driver-node-2jrfl\" (UID: \"1febd8fa-5641-449b-b1fd-c7d02603fd1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2jrfl" Nov 03 01:18:39.263183 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.262248 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d157814d-82d6-4af5-a648-c03ad35f3a41-etc-modprobe-d\") pod \"tuned-c6r2x\" (UID: \"d157814d-82d6-4af5-a648-c03ad35f3a41\") " pod="openshift-cluster-node-tuning-operator/tuned-c6r2x" Nov 03 01:18:39.263183 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.262271 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1febd8fa-5641-449b-b1fd-c7d02603fd1c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2jrfl\" (UID: \"1febd8fa-5641-449b-b1fd-c7d02603fd1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2jrfl" Nov 03 01:18:39.263183 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.262286 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8f478179-5984-48dd-b872-8f552954ecea-secret-grpc-tls\") pod \"thanos-querier-8bbf59f7f-cmcdf\" (UID: \"8f478179-5984-48dd-b872-8f552954ecea\") " pod="openshift-monitoring/thanos-querier-8bbf59f7f-cmcdf" Nov 03 01:18:39.263183 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.262301 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sq5fx\" (UniqueName: \"kubernetes.io/projected/30eede54-1970-4f52-b219-70d3c469ddd4-kube-api-access-sq5fx\") pod \"node-exporter-5z5ct\" (UID: \"30eede54-1970-4f52-b219-70d3c469ddd4\") " pod="openshift-monitoring/node-exporter-5z5ct" Nov 03 01:18:39.263183 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.262319 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/386397a9-5c8d-4221-894b-279e0b396afe-console-config\") pod \"console-69fcb6c6b8-jqgxm\" (UID: \"386397a9-5c8d-4221-894b-279e0b396afe\") " pod="openshift-console/console-69fcb6c6b8-jqgxm" Nov 03 01:18:39.263183 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.262329 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d157814d-82d6-4af5-a648-c03ad35f3a41-etc-modprobe-d\") pod \"tuned-c6r2x\" (UID: \"d157814d-82d6-4af5-a648-c03ad35f3a41\") " pod="openshift-cluster-node-tuning-operator/tuned-c6r2x" Nov 03 01:18:39.263183 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.262250 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e17d6c62-2952-420c-beda-49db831c48ac-host-var-lib-kubelet\") pod \"multus-44p7d\" (UID: \"e17d6c62-2952-420c-beda-49db831c48ac\") " pod="openshift-multus/multus-44p7d" Nov 03 01:18:39.263183 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.262341 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d157814d-82d6-4af5-a648-c03ad35f3a41-run\") pod \"tuned-c6r2x\" (UID: \"d157814d-82d6-4af5-a648-c03ad35f3a41\") " pod="openshift-cluster-node-tuning-operator/tuned-c6r2x" Nov 03 01:18:39.263183 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.262356 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1febd8fa-5641-449b-b1fd-c7d02603fd1c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2jrfl\" (UID: \"1febd8fa-5641-449b-b1fd-c7d02603fd1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2jrfl" Nov 03 01:18:39.263183 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.262398 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d157814d-82d6-4af5-a648-c03ad35f3a41-run\") pod \"tuned-c6r2x\" (UID: \"d157814d-82d6-4af5-a648-c03ad35f3a41\") " pod="openshift-cluster-node-tuning-operator/tuned-c6r2x" Nov 03 01:18:39.263183 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.262456 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8f478179-5984-48dd-b872-8f552954ecea-tmp\") pod \"thanos-querier-8bbf59f7f-cmcdf\" (UID: \"8f478179-5984-48dd-b872-8f552954ecea\") " pod="openshift-monitoring/thanos-querier-8bbf59f7f-cmcdf" Nov 03 01:18:39.263183 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.262484 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/386397a9-5c8d-4221-894b-279e0b396afe-console-serving-cert\") pod \"console-69fcb6c6b8-jqgxm\" (UID: \"386397a9-5c8d-4221-894b-279e0b396afe\") " pod="openshift-console/console-69fcb6c6b8-jqgxm" Nov 03 01:18:39.263183 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.262505 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/abdd79d9-8d87-4133-85a2-643d26fc9faa-host-run-netns\") pod \"ovnkube-node-n2cjg\" (UID: \"abdd79d9-8d87-4133-85a2-643d26fc9faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cjg" Nov 03 01:18:39.263183 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.262535 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/386397a9-5c8d-4221-894b-279e0b396afe-console-config\") pod \"console-69fcb6c6b8-jqgxm\" (UID: \"386397a9-5c8d-4221-894b-279e0b396afe\") " pod="openshift-console/console-69fcb6c6b8-jqgxm" Nov 03 01:18:39.263183 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.262550 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/abdd79d9-8d87-4133-85a2-643d26fc9faa-host-run-netns\") pod \"ovnkube-node-n2cjg\" (UID: \"abdd79d9-8d87-4133-85a2-643d26fc9faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cjg" Nov 03 01:18:39.263183 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.262546 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d157814d-82d6-4af5-a648-c03ad35f3a41-etc-sysctl-conf\") pod \"tuned-c6r2x\" (UID: \"d157814d-82d6-4af5-a648-c03ad35f3a41\") " pod="openshift-cluster-node-tuning-operator/tuned-c6r2x" Nov 03 01:18:39.263183 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.262572 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8f478179-5984-48dd-b872-8f552954ecea-tmp\") pod \"thanos-querier-8bbf59f7f-cmcdf\" (UID: \"8f478179-5984-48dd-b872-8f552954ecea\") " pod="openshift-monitoring/thanos-querier-8bbf59f7f-cmcdf" Nov 03 01:18:39.263183 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.262577 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8f478179-5984-48dd-b872-8f552954ecea-secret-grpc-tls\") pod \"thanos-querier-8bbf59f7f-cmcdf\" (UID: \"8f478179-5984-48dd-b872-8f552954ecea\") " pod="openshift-monitoring/thanos-querier-8bbf59f7f-cmcdf" Nov 03 01:18:39.263183 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.262575 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1febd8fa-5641-449b-b1fd-c7d02603fd1c-registration-dir\") pod \"aws-ebs-csi-driver-node-2jrfl\" (UID: \"1febd8fa-5641-449b-b1fd-c7d02603fd1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2jrfl" Nov 03 01:18:39.263183 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.262607 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1febd8fa-5641-449b-b1fd-c7d02603fd1c-registration-dir\") pod \"aws-ebs-csi-driver-node-2jrfl\" (UID: \"1febd8fa-5641-449b-b1fd-c7d02603fd1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2jrfl" Nov 03 01:18:39.263183 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.262608 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bff481c2-d05e-4b45-827d-f5c8a0f60d0b-host\") pod \"node-ca-jpcqz\" (UID: \"bff481c2-d05e-4b45-827d-f5c8a0f60d0b\") " pod="openshift-image-registry/node-ca-jpcqz" Nov 03 01:18:39.263183 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.262635 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bff481c2-d05e-4b45-827d-f5c8a0f60d0b-host\") pod \"node-ca-jpcqz\" (UID: \"bff481c2-d05e-4b45-827d-f5c8a0f60d0b\") " pod="openshift-image-registry/node-ca-jpcqz" Nov 03 01:18:39.263183 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.262637 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4c25011b-c85c-4736-9f72-842a96949072-system-cni-dir\") pod \"multus-additional-cni-plugins-jnvg6\" (UID: \"4c25011b-c85c-4736-9f72-842a96949072\") " pod="openshift-multus/multus-additional-cni-plugins-jnvg6" Nov 03 01:18:39.263183 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.262654 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4c25011b-c85c-4736-9f72-842a96949072-system-cni-dir\") pod \"multus-additional-cni-plugins-jnvg6\" (UID: \"4c25011b-c85c-4736-9f72-842a96949072\") " pod="openshift-multus/multus-additional-cni-plugins-jnvg6" Nov 03 01:18:39.263183 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.262666 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d157814d-82d6-4af5-a648-c03ad35f3a41-etc-sysctl-conf\") pod \"tuned-c6r2x\" (UID: \"d157814d-82d6-4af5-a648-c03ad35f3a41\") " pod="openshift-cluster-node-tuning-operator/tuned-c6r2x" Nov 03 01:18:39.263183 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.262672 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8f478179-5984-48dd-b872-8f552954ecea-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-8bbf59f7f-cmcdf\" (UID: \"8f478179-5984-48dd-b872-8f552954ecea\") " pod="openshift-monitoring/thanos-querier-8bbf59f7f-cmcdf" Nov 03 01:18:39.263183 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.262696 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fe434c5e-450b-4512-93aa-78d3e56e5002-agent-certs\") pod \"konnectivity-agent-dd9jt\" (UID: \"fe434c5e-450b-4512-93aa-78d3e56e5002\") " pod="kube-system/konnectivity-agent-dd9jt" Nov 03 01:18:39.263183 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.262712 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/386397a9-5c8d-4221-894b-279e0b396afe-console-oauth-config\") pod \"console-69fcb6c6b8-jqgxm\" (UID: \"386397a9-5c8d-4221-894b-279e0b396afe\") " pod="openshift-console/console-69fcb6c6b8-jqgxm" Nov 03 01:18:39.263183 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.262726 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wk6dw\" (UniqueName: \"kubernetes.io/projected/386397a9-5c8d-4221-894b-279e0b396afe-kube-api-access-wk6dw\") pod \"console-69fcb6c6b8-jqgxm\" (UID: \"386397a9-5c8d-4221-894b-279e0b396afe\") " pod="openshift-console/console-69fcb6c6b8-jqgxm" Nov 03 01:18:39.263183 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.262729 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/386397a9-5c8d-4221-894b-279e0b396afe-console-serving-cert\") pod \"console-69fcb6c6b8-jqgxm\" (UID: \"386397a9-5c8d-4221-894b-279e0b396afe\") " pod="openshift-console/console-69fcb6c6b8-jqgxm" Nov 03 01:18:39.263183 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.262743 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d157814d-82d6-4af5-a648-c03ad35f3a41-sys\") pod \"tuned-c6r2x\" (UID: \"d157814d-82d6-4af5-a648-c03ad35f3a41\") " pod="openshift-cluster-node-tuning-operator/tuned-c6r2x" Nov 03 01:18:39.263183 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.262767 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/30eede54-1970-4f52-b219-70d3c469ddd4-node-exporter-textfile\") pod \"node-exporter-5z5ct\" (UID: \"30eede54-1970-4f52-b219-70d3c469ddd4\") " pod="openshift-monitoring/node-exporter-5z5ct" Nov 03 01:18:39.263183 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.262790 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e17d6c62-2952-420c-beda-49db831c48ac-multus-socket-dir-parent\") pod \"multus-44p7d\" (UID: \"e17d6c62-2952-420c-beda-49db831c48ac\") " pod="openshift-multus/multus-44p7d" Nov 03 01:18:39.263183 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.262815 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/20152fc3-0b5f-4c58-94fb-e02094377c74-host-slash\") pod \"iptables-alerter-x4tvh\" (UID: \"20152fc3-0b5f-4c58-94fb-e02094377c74\") " pod="openshift-network-operator/iptables-alerter-x4tvh" Nov 03 01:18:39.263183 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.262863 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e17d6c62-2952-420c-beda-49db831c48ac-etc-kubernetes\") pod \"multus-44p7d\" (UID: \"e17d6c62-2952-420c-beda-49db831c48ac\") " pod="openshift-multus/multus-44p7d" Nov 03 01:18:39.263183 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.262873 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8f478179-5984-48dd-b872-8f552954ecea-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-8bbf59f7f-cmcdf\" (UID: \"8f478179-5984-48dd-b872-8f552954ecea\") " pod="openshift-monitoring/thanos-querier-8bbf59f7f-cmcdf" Nov 03 01:18:39.263183 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.262890 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1febd8fa-5641-449b-b1fd-c7d02603fd1c-sys-fs\") pod \"aws-ebs-csi-driver-node-2jrfl\" (UID: \"1febd8fa-5641-449b-b1fd-c7d02603fd1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2jrfl" Nov 03 01:18:39.263183 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.262867 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/386397a9-5c8d-4221-894b-279e0b396afe-console-oauth-config\") pod \"console-69fcb6c6b8-jqgxm\" (UID: \"386397a9-5c8d-4221-894b-279e0b396afe\") " pod="openshift-console/console-69fcb6c6b8-jqgxm" Nov 03 01:18:39.263183 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.262886 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/30eede54-1970-4f52-b219-70d3c469ddd4-node-exporter-textfile\") pod \"node-exporter-5z5ct\" (UID: \"30eede54-1970-4f52-b219-70d3c469ddd4\") " pod="openshift-monitoring/node-exporter-5z5ct" Nov 03 01:18:39.263183 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.262913 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/20152fc3-0b5f-4c58-94fb-e02094377c74-host-slash\") pod \"iptables-alerter-x4tvh\" (UID: \"20152fc3-0b5f-4c58-94fb-e02094377c74\") " pod="openshift-network-operator/iptables-alerter-x4tvh" Nov 03 01:18:39.263183 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.262890 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e17d6c62-2952-420c-beda-49db831c48ac-multus-socket-dir-parent\") pod \"multus-44p7d\" (UID: \"e17d6c62-2952-420c-beda-49db831c48ac\") " pod="openshift-multus/multus-44p7d" Nov 03 01:18:39.263183 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.262916 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjdgx\" (UniqueName: \"kubernetes.io/projected/9eb34322-124e-484d-a4fa-d79391ac36b1-kube-api-access-zjdgx\") pod \"monitoring-plugin-77f4f76594-ssqxs\" (UID: \"9eb34322-124e-484d-a4fa-d79391ac36b1\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-ssqxs" Nov 03 01:18:39.263183 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.262939 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e17d6c62-2952-420c-beda-49db831c48ac-etc-kubernetes\") pod \"multus-44p7d\" (UID: \"e17d6c62-2952-420c-beda-49db831c48ac\") " pod="openshift-multus/multus-44p7d" Nov 03 01:18:39.263183 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.262942 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fe434c5e-450b-4512-93aa-78d3e56e5002-agent-certs\") pod \"konnectivity-agent-dd9jt\" (UID: \"fe434c5e-450b-4512-93aa-78d3e56e5002\") " pod="kube-system/konnectivity-agent-dd9jt" Nov 03 01:18:39.263183 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.262951 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1febd8fa-5641-449b-b1fd-c7d02603fd1c-sys-fs\") pod \"aws-ebs-csi-driver-node-2jrfl\" (UID: \"1febd8fa-5641-449b-b1fd-c7d02603fd1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2jrfl" Nov 03 01:18:39.263183 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.262958 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/766a044f-fc66-4998-8006-343c0329860d-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-htfx6\" (UID: \"766a044f-fc66-4998-8006-343c0329860d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-htfx6" Nov 03 01:18:39.263183 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.262980 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d157814d-82d6-4af5-a648-c03ad35f3a41-sys\") pod \"tuned-c6r2x\" (UID: \"d157814d-82d6-4af5-a648-c03ad35f3a41\") " pod="openshift-cluster-node-tuning-operator/tuned-c6r2x" Nov 03 01:18:39.263183 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.262990 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7ckb\" (UniqueName: \"kubernetes.io/projected/766a044f-fc66-4998-8006-343c0329860d-kube-api-access-b7ckb\") pod \"kube-state-metrics-5dc8d7d89b-htfx6\" (UID: \"766a044f-fc66-4998-8006-343c0329860d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-htfx6" Nov 03 01:18:39.263183 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.263008 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0e35c4cd-ef18-40c1-8568-f77773e378b2-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-lcwkf\" (UID: \"0e35c4cd-ef18-40c1-8568-f77773e378b2\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lcwkf" Nov 03 01:18:39.263183 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.263024 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shwrh\" (UniqueName: \"kubernetes.io/projected/0e35c4cd-ef18-40c1-8568-f77773e378b2-kube-api-access-shwrh\") pod \"openshift-state-metrics-7bd7489957-lcwkf\" (UID: \"0e35c4cd-ef18-40c1-8568-f77773e378b2\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lcwkf" Nov 03 01:18:39.263183 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.263039 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/30eede54-1970-4f52-b219-70d3c469ddd4-metrics-client-ca\") pod \"node-exporter-5z5ct\" (UID: \"30eede54-1970-4f52-b219-70d3c469ddd4\") " pod="openshift-monitoring/node-exporter-5z5ct" Nov 03 01:18:39.263183 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.263060 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/abdd79d9-8d87-4133-85a2-643d26fc9faa-host-cni-netd\") pod \"ovnkube-node-n2cjg\" (UID: \"abdd79d9-8d87-4133-85a2-643d26fc9faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cjg" Nov 03 01:18:39.263183 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.263076 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e17d6c62-2952-420c-beda-49db831c48ac-host-run-k8s-cni-cncf-io\") pod \"multus-44p7d\" (UID: \"e17d6c62-2952-420c-beda-49db831c48ac\") " pod="openshift-multus/multus-44p7d" Nov 03 01:18:39.263183 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.263104 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e17d6c62-2952-420c-beda-49db831c48ac-host-run-k8s-cni-cncf-io\") pod \"multus-44p7d\" (UID: \"e17d6c62-2952-420c-beda-49db831c48ac\") " pod="openshift-multus/multus-44p7d" Nov 03 01:18:39.263183 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.263172 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/766a044f-fc66-4998-8006-343c0329860d-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-htfx6\" (UID: \"766a044f-fc66-4998-8006-343c0329860d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-htfx6" Nov 03 01:18:39.263183 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.263176 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/abdd79d9-8d87-4133-85a2-643d26fc9faa-host-cni-netd\") pod \"ovnkube-node-n2cjg\" (UID: \"abdd79d9-8d87-4133-85a2-643d26fc9faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cjg" Nov 03 01:18:39.263183 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.263201 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d157814d-82d6-4af5-a648-c03ad35f3a41-etc-kubernetes\") pod \"tuned-c6r2x\" (UID: \"d157814d-82d6-4af5-a648-c03ad35f3a41\") " pod="openshift-cluster-node-tuning-operator/tuned-c6r2x" Nov 03 01:18:39.263183 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.263227 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2320cce9-0fdb-470a-ae01-e88118ba0419-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bbfdw\" (UID: \"2320cce9-0fdb-470a-ae01-e88118ba0419\") " pod="openshift-insights/insights-runtime-extractor-bbfdw" Nov 03 01:18:39.265718 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.263246 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/386397a9-5c8d-4221-894b-279e0b396afe-tmp\") pod \"console-69fcb6c6b8-jqgxm\" (UID: \"386397a9-5c8d-4221-894b-279e0b396afe\") " pod="openshift-console/console-69fcb6c6b8-jqgxm" Nov 03 01:18:39.265718 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.263245 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d157814d-82d6-4af5-a648-c03ad35f3a41-etc-kubernetes\") pod \"tuned-c6r2x\" (UID: \"d157814d-82d6-4af5-a648-c03ad35f3a41\") " pod="openshift-cluster-node-tuning-operator/tuned-c6r2x" Nov 03 01:18:39.265718 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.263261 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0e35c4cd-ef18-40c1-8568-f77773e378b2-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-lcwkf\" (UID: \"0e35c4cd-ef18-40c1-8568-f77773e378b2\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lcwkf" Nov 03 01:18:39.265718 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.263268 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/30eede54-1970-4f52-b219-70d3c469ddd4-metrics-client-ca\") pod \"node-exporter-5z5ct\" (UID: \"30eede54-1970-4f52-b219-70d3c469ddd4\") " pod="openshift-monitoring/node-exporter-5z5ct" Nov 03 01:18:39.265718 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.263278 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/386397a9-5c8d-4221-894b-279e0b396afe-trusted-ca-bundle\") pod \"console-69fcb6c6b8-jqgxm\" (UID: \"386397a9-5c8d-4221-894b-279e0b396afe\") " pod="openshift-console/console-69fcb6c6b8-jqgxm" Nov 03 01:18:39.265718 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.263404 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/386397a9-5c8d-4221-894b-279e0b396afe-tmp\") pod \"console-69fcb6c6b8-jqgxm\" (UID: \"386397a9-5c8d-4221-894b-279e0b396afe\") " pod="openshift-console/console-69fcb6c6b8-jqgxm" Nov 03 01:18:39.265718 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.263403 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/abdd79d9-8d87-4133-85a2-643d26fc9faa-host-kubelet\") pod \"ovnkube-node-n2cjg\" (UID: \"abdd79d9-8d87-4133-85a2-643d26fc9faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cjg" Nov 03 01:18:39.265718 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.263439 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d157814d-82d6-4af5-a648-c03ad35f3a41-etc-sysconfig\") pod \"tuned-c6r2x\" (UID: \"d157814d-82d6-4af5-a648-c03ad35f3a41\") " pod="openshift-cluster-node-tuning-operator/tuned-c6r2x" Nov 03 01:18:39.265718 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.263442 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/abdd79d9-8d87-4133-85a2-643d26fc9faa-host-kubelet\") pod \"ovnkube-node-n2cjg\" (UID: \"abdd79d9-8d87-4133-85a2-643d26fc9faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cjg" Nov 03 01:18:39.265718 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.263456 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/8f478179-5984-48dd-b872-8f552954ecea-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-8bbf59f7f-cmcdf\" (UID: \"8f478179-5984-48dd-b872-8f552954ecea\") " pod="openshift-monitoring/thanos-querier-8bbf59f7f-cmcdf" Nov 03 01:18:39.265718 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.263472 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2320cce9-0fdb-470a-ae01-e88118ba0419-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bbfdw\" (UID: \"2320cce9-0fdb-470a-ae01-e88118ba0419\") " pod="openshift-insights/insights-runtime-extractor-bbfdw" Nov 03 01:18:39.265718 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.263489 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d157814d-82d6-4af5-a648-c03ad35f3a41-etc-sysconfig\") pod \"tuned-c6r2x\" (UID: \"d157814d-82d6-4af5-a648-c03ad35f3a41\") " pod="openshift-cluster-node-tuning-operator/tuned-c6r2x" Nov 03 01:18:39.265718 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.263485 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/30eede54-1970-4f52-b219-70d3c469ddd4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5z5ct\" (UID: \"30eede54-1970-4f52-b219-70d3c469ddd4\") " pod="openshift-monitoring/node-exporter-5z5ct" Nov 03 01:18:39.265718 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.263528 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e17d6c62-2952-420c-beda-49db831c48ac-system-cni-dir\") pod \"multus-44p7d\" (UID: \"e17d6c62-2952-420c-beda-49db831c48ac\") " pod="openshift-multus/multus-44p7d" Nov 03 01:18:39.265718 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.263551 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2320cce9-0fdb-470a-ae01-e88118ba0419-data-volume\") pod \"insights-runtime-extractor-bbfdw\" (UID: \"2320cce9-0fdb-470a-ae01-e88118ba0419\") " pod="openshift-insights/insights-runtime-extractor-bbfdw" Nov 03 01:18:39.265718 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.263576 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/abdd79d9-8d87-4133-85a2-643d26fc9faa-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-n2cjg\" (UID: \"abdd79d9-8d87-4133-85a2-643d26fc9faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cjg" Nov 03 01:18:39.265718 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.263607 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e17d6c62-2952-420c-beda-49db831c48ac-system-cni-dir\") pod \"multus-44p7d\" (UID: \"e17d6c62-2952-420c-beda-49db831c48ac\") " pod="openshift-multus/multus-44p7d" Nov 03 01:18:39.265718 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.263616 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/386397a9-5c8d-4221-894b-279e0b396afe-trusted-ca-bundle\") pod \"console-69fcb6c6b8-jqgxm\" (UID: \"386397a9-5c8d-4221-894b-279e0b396afe\") " pod="openshift-console/console-69fcb6c6b8-jqgxm" Nov 03 01:18:39.265718 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.263618 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6179266f-07ff-48cd-bf03-3f1e961529d7-dbus\") pod \"global-pull-secret-syncer-dfr9t\" (UID: \"6179266f-07ff-48cd-bf03-3f1e961529d7\") " pod="kube-system/global-pull-secret-syncer-dfr9t" Nov 03 01:18:39.265718 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.263633 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/8f478179-5984-48dd-b872-8f552954ecea-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-8bbf59f7f-cmcdf\" (UID: \"8f478179-5984-48dd-b872-8f552954ecea\") " pod="openshift-monitoring/thanos-querier-8bbf59f7f-cmcdf" Nov 03 01:18:39.265718 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.263658 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/abdd79d9-8d87-4133-85a2-643d26fc9faa-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-n2cjg\" (UID: \"abdd79d9-8d87-4133-85a2-643d26fc9faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cjg" Nov 03 01:18:39.265718 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.263665 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/abdd79d9-8d87-4133-85a2-643d26fc9faa-log-socket\") pod \"ovnkube-node-n2cjg\" (UID: \"abdd79d9-8d87-4133-85a2-643d26fc9faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cjg" Nov 03 01:18:39.265718 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.263687 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/abdd79d9-8d87-4133-85a2-643d26fc9faa-host-cni-bin\") pod \"ovnkube-node-n2cjg\" (UID: \"abdd79d9-8d87-4133-85a2-643d26fc9faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cjg" Nov 03 01:18:39.265718 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.263690 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/abdd79d9-8d87-4133-85a2-643d26fc9faa-log-socket\") pod \"ovnkube-node-n2cjg\" (UID: \"abdd79d9-8d87-4133-85a2-643d26fc9faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cjg" Nov 03 01:18:39.265718 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.263665 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6179266f-07ff-48cd-bf03-3f1e961529d7-dbus\") pod \"global-pull-secret-syncer-dfr9t\" (UID: \"6179266f-07ff-48cd-bf03-3f1e961529d7\") " pod="kube-system/global-pull-secret-syncer-dfr9t" Nov 03 01:18:39.265718 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.263688 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2320cce9-0fdb-470a-ae01-e88118ba0419-data-volume\") pod \"insights-runtime-extractor-bbfdw\" (UID: \"2320cce9-0fdb-470a-ae01-e88118ba0419\") " pod="openshift-insights/insights-runtime-extractor-bbfdw" Nov 03 01:18:39.265718 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.263702 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4c25011b-c85c-4736-9f72-842a96949072-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jnvg6\" (UID: \"4c25011b-c85c-4736-9f72-842a96949072\") " pod="openshift-multus/multus-additional-cni-plugins-jnvg6" Nov 03 01:18:39.265718 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.263688 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/30eede54-1970-4f52-b219-70d3c469ddd4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5z5ct\" (UID: \"30eede54-1970-4f52-b219-70d3c469ddd4\") " pod="openshift-monitoring/node-exporter-5z5ct" Nov 03 01:18:39.265718 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.263732 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/abdd79d9-8d87-4133-85a2-643d26fc9faa-host-cni-bin\") pod \"ovnkube-node-n2cjg\" (UID: \"abdd79d9-8d87-4133-85a2-643d26fc9faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cjg" Nov 03 01:18:39.265718 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.263735 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4c25011b-c85c-4736-9f72-842a96949072-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jnvg6\" (UID: \"4c25011b-c85c-4736-9f72-842a96949072\") " pod="openshift-multus/multus-additional-cni-plugins-jnvg6" Nov 03 01:18:39.265718 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.263761 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/766a044f-fc66-4998-8006-343c0329860d-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-htfx6\" (UID: \"766a044f-fc66-4998-8006-343c0329860d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-htfx6" Nov 03 01:18:39.265718 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.263785 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0e35c4cd-ef18-40c1-8568-f77773e378b2-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-lcwkf\" (UID: \"0e35c4cd-ef18-40c1-8568-f77773e378b2\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lcwkf" Nov 03 01:18:39.265718 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.263802 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0e35c4cd-ef18-40c1-8568-f77773e378b2-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-lcwkf\" (UID: \"0e35c4cd-ef18-40c1-8568-f77773e378b2\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lcwkf" Nov 03 01:18:39.265718 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.263821 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e17d6c62-2952-420c-beda-49db831c48ac-host-var-lib-cni-bin\") pod \"multus-44p7d\" (UID: \"e17d6c62-2952-420c-beda-49db831c48ac\") " pod="openshift-multus/multus-44p7d" Nov 03 01:18:39.265718 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.263841 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fdb6f2d-9138-4f57-af12-c3909dff6e5f-tmp-dir\") pod \"dns-default-kmpbz\" (UID: \"1fdb6f2d-9138-4f57-af12-c3909dff6e5f\") " pod="openshift-dns/dns-default-kmpbz" Nov 03 01:18:39.265718 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.263859 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/766a044f-fc66-4998-8006-343c0329860d-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-htfx6\" (UID: \"766a044f-fc66-4998-8006-343c0329860d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-htfx6" Nov 03 01:18:39.265718 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.263883 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8l47\" (UniqueName: \"kubernetes.io/projected/8f478179-5984-48dd-b872-8f552954ecea-kube-api-access-v8l47\") pod \"thanos-querier-8bbf59f7f-cmcdf\" (UID: \"8f478179-5984-48dd-b872-8f552954ecea\") " pod="openshift-monitoring/thanos-querier-8bbf59f7f-cmcdf" Nov 03 01:18:39.265718 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.263908 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/30eede54-1970-4f52-b219-70d3c469ddd4-node-exporter-tls\") pod \"node-exporter-5z5ct\" (UID: \"30eede54-1970-4f52-b219-70d3c469ddd4\") " pod="openshift-monitoring/node-exporter-5z5ct" Nov 03 01:18:39.265718 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.263910 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e17d6c62-2952-420c-beda-49db831c48ac-host-var-lib-cni-bin\") pod \"multus-44p7d\" (UID: \"e17d6c62-2952-420c-beda-49db831c48ac\") " pod="openshift-multus/multus-44p7d" Nov 03 01:18:39.265718 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.263934 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/766a044f-fc66-4998-8006-343c0329860d-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-htfx6\" (UID: \"766a044f-fc66-4998-8006-343c0329860d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-htfx6" Nov 03 01:18:39.265718 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.263955 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/386397a9-5c8d-4221-894b-279e0b396afe-service-ca\") pod \"console-69fcb6c6b8-jqgxm\" (UID: \"386397a9-5c8d-4221-894b-279e0b396afe\") " pod="openshift-console/console-69fcb6c6b8-jqgxm" Nov 03 01:18:39.265718 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.263969 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0e35c4cd-ef18-40c1-8568-f77773e378b2-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-lcwkf\" (UID: \"0e35c4cd-ef18-40c1-8568-f77773e378b2\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lcwkf" Nov 03 01:18:39.265718 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.263983 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/abdd79d9-8d87-4133-85a2-643d26fc9faa-systemd-units\") pod \"ovnkube-node-n2cjg\" (UID: \"abdd79d9-8d87-4133-85a2-643d26fc9faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cjg" Nov 03 01:18:39.265718 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.263957 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fdb6f2d-9138-4f57-af12-c3909dff6e5f-tmp-dir\") pod \"dns-default-kmpbz\" (UID: \"1fdb6f2d-9138-4f57-af12-c3909dff6e5f\") " pod="openshift-dns/dns-default-kmpbz" Nov 03 01:18:39.265718 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.264025 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/8f478179-5984-48dd-b872-8f552954ecea-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-8bbf59f7f-cmcdf\" (UID: \"8f478179-5984-48dd-b872-8f552954ecea\") " pod="openshift-monitoring/thanos-querier-8bbf59f7f-cmcdf" Nov 03 01:18:39.265718 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.264051 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/abdd79d9-8d87-4133-85a2-643d26fc9faa-systemd-units\") pod \"ovnkube-node-n2cjg\" (UID: \"abdd79d9-8d87-4133-85a2-643d26fc9faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cjg" Nov 03 01:18:39.265718 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.264055 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0e35c4cd-ef18-40c1-8568-f77773e378b2-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-lcwkf\" (UID: \"0e35c4cd-ef18-40c1-8568-f77773e378b2\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lcwkf" Nov 03 01:18:39.265718 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.264055 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/30eede54-1970-4f52-b219-70d3c469ddd4-root\") pod \"node-exporter-5z5ct\" (UID: \"30eede54-1970-4f52-b219-70d3c469ddd4\") " pod="openshift-monitoring/node-exporter-5z5ct" Nov 03 01:18:39.265718 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.264076 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/766a044f-fc66-4998-8006-343c0329860d-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-htfx6\" (UID: \"766a044f-fc66-4998-8006-343c0329860d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-htfx6" Nov 03 01:18:39.265718 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.264114 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/386397a9-5c8d-4221-894b-279e0b396afe-service-ca\") pod \"console-69fcb6c6b8-jqgxm\" (UID: \"386397a9-5c8d-4221-894b-279e0b396afe\") " pod="openshift-console/console-69fcb6c6b8-jqgxm" Nov 03 01:18:39.265718 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.264139 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/30eede54-1970-4f52-b219-70d3c469ddd4-node-exporter-tls\") pod \"node-exporter-5z5ct\" (UID: \"30eede54-1970-4f52-b219-70d3c469ddd4\") " pod="openshift-monitoring/node-exporter-5z5ct" Nov 03 01:18:39.265718 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.264188 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e17d6c62-2952-420c-beda-49db831c48ac-multus-cni-dir\") pod \"multus-44p7d\" (UID: \"e17d6c62-2952-420c-beda-49db831c48ac\") " pod="openshift-multus/multus-44p7d" Nov 03 01:18:39.265718 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.264211 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d157814d-82d6-4af5-a648-c03ad35f3a41-etc-systemd\") pod \"tuned-c6r2x\" (UID: \"d157814d-82d6-4af5-a648-c03ad35f3a41\") " pod="openshift-cluster-node-tuning-operator/tuned-c6r2x" Nov 03 01:18:39.265718 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.264227 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4c25011b-c85c-4736-9f72-842a96949072-cnibin\") pod \"multus-additional-cni-plugins-jnvg6\" (UID: \"4c25011b-c85c-4736-9f72-842a96949072\") " pod="openshift-multus/multus-additional-cni-plugins-jnvg6" Nov 03 01:18:39.265718 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.264243 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/97cb83aa-8922-4221-8407-fbbb6c39f5c7-cert\") pod \"ingress-canary-4vr4n\" (UID: \"97cb83aa-8922-4221-8407-fbbb6c39f5c7\") " pod="openshift-ingress-canary/ingress-canary-4vr4n" Nov 03 01:18:39.265718 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.264241 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/8f478179-5984-48dd-b872-8f552954ecea-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-8bbf59f7f-cmcdf\" (UID: \"8f478179-5984-48dd-b872-8f552954ecea\") " pod="openshift-monitoring/thanos-querier-8bbf59f7f-cmcdf" Nov 03 01:18:39.265718 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.264265 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1fdb6f2d-9138-4f57-af12-c3909dff6e5f-metrics-tls\") pod \"dns-default-kmpbz\" (UID: \"1fdb6f2d-9138-4f57-af12-c3909dff6e5f\") " pod="openshift-dns/dns-default-kmpbz" Nov 03 01:18:39.265718 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.264287 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d157814d-82d6-4af5-a648-c03ad35f3a41-etc-systemd\") pod \"tuned-c6r2x\" (UID: \"d157814d-82d6-4af5-a648-c03ad35f3a41\") " pod="openshift-cluster-node-tuning-operator/tuned-c6r2x" Nov 03 01:18:39.265718 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.264296 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4c25011b-c85c-4736-9f72-842a96949072-cnibin\") pod \"multus-additional-cni-plugins-jnvg6\" (UID: \"4c25011b-c85c-4736-9f72-842a96949072\") " pod="openshift-multus/multus-additional-cni-plugins-jnvg6" Nov 03 01:18:39.265718 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.264274 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e17d6c62-2952-420c-beda-49db831c48ac-multus-cni-dir\") pod \"multus-44p7d\" (UID: \"e17d6c62-2952-420c-beda-49db831c48ac\") " pod="openshift-multus/multus-44p7d" Nov 03 01:18:39.265718 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.264312 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8f478179-5984-48dd-b872-8f552954ecea-metrics-client-ca\") pod \"thanos-querier-8bbf59f7f-cmcdf\" (UID: \"8f478179-5984-48dd-b872-8f552954ecea\") " pod="openshift-monitoring/thanos-querier-8bbf59f7f-cmcdf" Nov 03 01:18:39.265718 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.264329 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e17d6c62-2952-420c-beda-49db831c48ac-cnibin\") pod \"multus-44p7d\" (UID: \"e17d6c62-2952-420c-beda-49db831c48ac\") " pod="openshift-multus/multus-44p7d" Nov 03 01:18:39.265718 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.264354 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4v9cn\" (UniqueName: \"kubernetes.io/projected/97cb83aa-8922-4221-8407-fbbb6c39f5c7-kube-api-access-4v9cn\") pod \"ingress-canary-4vr4n\" (UID: \"97cb83aa-8922-4221-8407-fbbb6c39f5c7\") " pod="openshift-ingress-canary/ingress-canary-4vr4n" Nov 03 01:18:39.265718 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.264388 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8flcc\" (UniqueName: \"kubernetes.io/projected/1fdb6f2d-9138-4f57-af12-c3909dff6e5f-kube-api-access-8flcc\") pod \"dns-default-kmpbz\" (UID: \"1fdb6f2d-9138-4f57-af12-c3909dff6e5f\") " pod="openshift-dns/dns-default-kmpbz" Nov 03 01:18:39.265718 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.264406 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e17d6c62-2952-420c-beda-49db831c48ac-cnibin\") pod \"multus-44p7d\" (UID: \"e17d6c62-2952-420c-beda-49db831c48ac\") " pod="openshift-multus/multus-44p7d" Nov 03 01:18:39.265718 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.264407 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/abdd79d9-8d87-4133-85a2-643d26fc9faa-var-lib-openvswitch\") pod \"ovnkube-node-n2cjg\" (UID: \"abdd79d9-8d87-4133-85a2-643d26fc9faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cjg" Nov 03 01:18:39.265718 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.264433 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/abdd79d9-8d87-4133-85a2-643d26fc9faa-var-lib-openvswitch\") pod \"ovnkube-node-n2cjg\" (UID: \"abdd79d9-8d87-4133-85a2-643d26fc9faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cjg" Nov 03 01:18:39.265718 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.264439 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9eb34322-124e-484d-a4fa-d79391ac36b1-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-ssqxs\" (UID: \"9eb34322-124e-484d-a4fa-d79391ac36b1\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-ssqxs" Nov 03 01:18:39.265718 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.264465 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/97cb83aa-8922-4221-8407-fbbb6c39f5c7-cert\") pod \"ingress-canary-4vr4n\" (UID: \"97cb83aa-8922-4221-8407-fbbb6c39f5c7\") " pod="openshift-ingress-canary/ingress-canary-4vr4n" Nov 03 01:18:39.265718 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.264507 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1fdb6f2d-9138-4f57-af12-c3909dff6e5f-metrics-tls\") pod \"dns-default-kmpbz\" (UID: \"1fdb6f2d-9138-4f57-af12-c3909dff6e5f\") " pod="openshift-dns/dns-default-kmpbz" Nov 03 01:18:39.265718 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.264518 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8f478179-5984-48dd-b872-8f552954ecea-metrics-client-ca\") pod \"thanos-querier-8bbf59f7f-cmcdf\" (UID: \"8f478179-5984-48dd-b872-8f552954ecea\") " pod="openshift-monitoring/thanos-querier-8bbf59f7f-cmcdf" Nov 03 01:18:39.265718 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.264534 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d157814d-82d6-4af5-a648-c03ad35f3a41-lib-modules\") pod \"tuned-c6r2x\" (UID: \"d157814d-82d6-4af5-a648-c03ad35f3a41\") " pod="openshift-cluster-node-tuning-operator/tuned-c6r2x" Nov 03 01:18:39.265718 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.264563 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/30eede54-1970-4f52-b219-70d3c469ddd4-sys\") pod \"node-exporter-5z5ct\" (UID: \"30eede54-1970-4f52-b219-70d3c469ddd4\") " pod="openshift-monitoring/node-exporter-5z5ct" Nov 03 01:18:39.265718 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.264590 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/abdd79d9-8d87-4133-85a2-643d26fc9faa-run-openvswitch\") pod \"ovnkube-node-n2cjg\" (UID: \"abdd79d9-8d87-4133-85a2-643d26fc9faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cjg" Nov 03 01:18:39.265718 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.264634 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1febd8fa-5641-449b-b1fd-c7d02603fd1c-socket-dir\") pod \"aws-ebs-csi-driver-node-2jrfl\" (UID: \"1febd8fa-5641-449b-b1fd-c7d02603fd1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2jrfl" Nov 03 01:18:39.265718 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.264640 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d157814d-82d6-4af5-a648-c03ad35f3a41-lib-modules\") pod \"tuned-c6r2x\" (UID: \"d157814d-82d6-4af5-a648-c03ad35f3a41\") " pod="openshift-cluster-node-tuning-operator/tuned-c6r2x" Nov 03 01:18:39.265718 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.264652 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/386397a9-5c8d-4221-894b-279e0b396afe-oauth-serving-cert\") pod \"console-69fcb6c6b8-jqgxm\" (UID: \"386397a9-5c8d-4221-894b-279e0b396afe\") " pod="openshift-console/console-69fcb6c6b8-jqgxm" Nov 03 01:18:39.265718 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.264656 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/abdd79d9-8d87-4133-85a2-643d26fc9faa-run-openvswitch\") pod \"ovnkube-node-n2cjg\" (UID: \"abdd79d9-8d87-4133-85a2-643d26fc9faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cjg" Nov 03 01:18:39.265718 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.264663 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9eb34322-124e-484d-a4fa-d79391ac36b1-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-ssqxs\" (UID: \"9eb34322-124e-484d-a4fa-d79391ac36b1\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-ssqxs" Nov 03 01:18:39.265718 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.264667 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/abdd79d9-8d87-4133-85a2-643d26fc9faa-host-run-ovn-kubernetes\") pod \"ovnkube-node-n2cjg\" (UID: \"abdd79d9-8d87-4133-85a2-643d26fc9faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cjg" Nov 03 01:18:39.265718 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.264688 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1febd8fa-5641-449b-b1fd-c7d02603fd1c-socket-dir\") pod \"aws-ebs-csi-driver-node-2jrfl\" (UID: \"1febd8fa-5641-449b-b1fd-c7d02603fd1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2jrfl" Nov 03 01:18:39.265718 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.264706 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/abdd79d9-8d87-4133-85a2-643d26fc9faa-host-run-ovn-kubernetes\") pod \"ovnkube-node-n2cjg\" (UID: \"abdd79d9-8d87-4133-85a2-643d26fc9faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cjg" Nov 03 01:18:39.265718 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.264723 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2320cce9-0fdb-470a-ae01-e88118ba0419-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bbfdw\" (UID: \"2320cce9-0fdb-470a-ae01-e88118ba0419\") " pod="openshift-insights/insights-runtime-extractor-bbfdw" Nov 03 01:18:39.265718 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.264753 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8f478179-5984-48dd-b872-8f552954ecea-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-8bbf59f7f-cmcdf\" (UID: \"8f478179-5984-48dd-b872-8f552954ecea\") " pod="openshift-monitoring/thanos-querier-8bbf59f7f-cmcdf" Nov 03 01:18:39.265718 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.264782 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/30eede54-1970-4f52-b219-70d3c469ddd4-node-exporter-wtmp\") pod \"node-exporter-5z5ct\" (UID: \"30eede54-1970-4f52-b219-70d3c469ddd4\") " pod="openshift-monitoring/node-exporter-5z5ct" Nov 03 01:18:39.265718 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.264811 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/30eede54-1970-4f52-b219-70d3c469ddd4-node-exporter-accelerators-collector-config\") pod \"node-exporter-5z5ct\" (UID: \"30eede54-1970-4f52-b219-70d3c469ddd4\") " pod="openshift-monitoring/node-exporter-5z5ct" Nov 03 01:18:39.265718 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.264820 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/386397a9-5c8d-4221-894b-279e0b396afe-oauth-serving-cert\") pod \"console-69fcb6c6b8-jqgxm\" (UID: \"386397a9-5c8d-4221-894b-279e0b396afe\") " pod="openshift-console/console-69fcb6c6b8-jqgxm" Nov 03 01:18:39.265718 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.264837 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/abdd79d9-8d87-4133-85a2-643d26fc9faa-etc-openvswitch\") pod \"ovnkube-node-n2cjg\" (UID: \"abdd79d9-8d87-4133-85a2-643d26fc9faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cjg" Nov 03 01:18:39.265718 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.264886 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/abdd79d9-8d87-4133-85a2-643d26fc9faa-etc-openvswitch\") pod \"ovnkube-node-n2cjg\" (UID: \"abdd79d9-8d87-4133-85a2-643d26fc9faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2cjg" Nov 03 01:18:39.265718 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.264897 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2320cce9-0fdb-470a-ae01-e88118ba0419-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bbfdw\" (UID: \"2320cce9-0fdb-470a-ae01-e88118ba0419\") " pod="openshift-insights/insights-runtime-extractor-bbfdw" Nov 03 01:18:39.265718 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.264897 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/766a044f-fc66-4998-8006-343c0329860d-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-htfx6\" (UID: \"766a044f-fc66-4998-8006-343c0329860d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-htfx6" Nov 03 01:18:39.265718 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.264935 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/acada406-fef9-4a03-ae21-c2351ef0a4cf-hosts-file\") pod \"node-resolver-5qr9g\" (UID: \"acada406-fef9-4a03-ae21-c2351ef0a4cf\") " pod="openshift-dns/node-resolver-5qr9g" Nov 03 01:18:39.265718 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.264970 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8f478179-5984-48dd-b872-8f552954ecea-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-8bbf59f7f-cmcdf\" (UID: \"8f478179-5984-48dd-b872-8f552954ecea\") " pod="openshift-monitoring/thanos-querier-8bbf59f7f-cmcdf" Nov 03 01:18:39.265718 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.264989 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/acada406-fef9-4a03-ae21-c2351ef0a4cf-hosts-file\") pod \"node-resolver-5qr9g\" (UID: \"acada406-fef9-4a03-ae21-c2351ef0a4cf\") " pod="openshift-dns/node-resolver-5qr9g" Nov 03 01:18:39.265718 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.265068 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/30eede54-1970-4f52-b219-70d3c469ddd4-node-exporter-accelerators-collector-config\") pod \"node-exporter-5z5ct\" (UID: \"30eede54-1970-4f52-b219-70d3c469ddd4\") " pod="openshift-monitoring/node-exporter-5z5ct" Nov 03 01:18:39.265718 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.265095 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/766a044f-fc66-4998-8006-343c0329860d-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-htfx6\" (UID: \"766a044f-fc66-4998-8006-343c0329860d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-htfx6" Nov 03 01:18:39.322229 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.322195 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2cz9\" (UniqueName: \"kubernetes.io/projected/2320cce9-0fdb-470a-ae01-e88118ba0419-kube-api-access-b2cz9\") pod \"insights-runtime-extractor-bbfdw\" (UID: \"2320cce9-0fdb-470a-ae01-e88118ba0419\") " pod="openshift-insights/insights-runtime-extractor-bbfdw" Nov 03 01:18:39.322450 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.322432 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sq5fx\" (UniqueName: \"kubernetes.io/projected/30eede54-1970-4f52-b219-70d3c469ddd4-kube-api-access-sq5fx\") pod \"node-exporter-5z5ct\" (UID: \"30eede54-1970-4f52-b219-70d3c469ddd4\") " pod="openshift-monitoring/node-exporter-5z5ct" Nov 03 01:18:39.323691 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.323664 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wk6dw\" (UniqueName: \"kubernetes.io/projected/386397a9-5c8d-4221-894b-279e0b396afe-kube-api-access-wk6dw\") pod \"console-69fcb6c6b8-jqgxm\" (UID: \"386397a9-5c8d-4221-894b-279e0b396afe\") " pod="openshift-console/console-69fcb6c6b8-jqgxm" Nov 03 01:18:39.323806 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.323714 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shwrh\" (UniqueName: \"kubernetes.io/projected/0e35c4cd-ef18-40c1-8568-f77773e378b2-kube-api-access-shwrh\") pod \"openshift-state-metrics-7bd7489957-lcwkf\" (UID: \"0e35c4cd-ef18-40c1-8568-f77773e378b2\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lcwkf" Nov 03 01:18:39.323870 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.323849 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjdgx\" (UniqueName: \"kubernetes.io/projected/9eb34322-124e-484d-a4fa-d79391ac36b1-kube-api-access-zjdgx\") pod \"monitoring-plugin-77f4f76594-ssqxs\" (UID: \"9eb34322-124e-484d-a4fa-d79391ac36b1\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-ssqxs" Nov 03 01:18:39.323954 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.323940 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7ckb\" (UniqueName: \"kubernetes.io/projected/766a044f-fc66-4998-8006-343c0329860d-kube-api-access-b7ckb\") pod \"kube-state-metrics-5dc8d7d89b-htfx6\" (UID: \"766a044f-fc66-4998-8006-343c0329860d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-htfx6" Nov 03 01:18:39.324288 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.324270 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8l47\" (UniqueName: \"kubernetes.io/projected/8f478179-5984-48dd-b872-8f552954ecea-kube-api-access-v8l47\") pod \"thanos-querier-8bbf59f7f-cmcdf\" (UID: \"8f478179-5984-48dd-b872-8f552954ecea\") " pod="openshift-monitoring/thanos-querier-8bbf59f7f-cmcdf" Nov 03 01:18:39.324762 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.324746 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8flcc\" (UniqueName: \"kubernetes.io/projected/1fdb6f2d-9138-4f57-af12-c3909dff6e5f-kube-api-access-8flcc\") pod \"dns-default-kmpbz\" (UID: \"1fdb6f2d-9138-4f57-af12-c3909dff6e5f\") " pod="openshift-dns/dns-default-kmpbz" Nov 03 01:18:39.324841 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.324825 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4v9cn\" (UniqueName: \"kubernetes.io/projected/97cb83aa-8922-4221-8407-fbbb6c39f5c7-kube-api-access-4v9cn\") pod \"ingress-canary-4vr4n\" (UID: \"97cb83aa-8922-4221-8407-fbbb6c39f5c7\") " pod="openshift-ingress-canary/ingress-canary-4vr4n" Nov 03 01:18:39.365880 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.365855 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/30eede54-1970-4f52-b219-70d3c469ddd4-root\") pod \"node-exporter-5z5ct\" (UID: \"30eede54-1970-4f52-b219-70d3c469ddd4\") " pod="openshift-monitoring/node-exporter-5z5ct" Nov 03 01:18:39.365965 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.365887 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/30eede54-1970-4f52-b219-70d3c469ddd4-sys\") pod \"node-exporter-5z5ct\" (UID: \"30eede54-1970-4f52-b219-70d3c469ddd4\") " pod="openshift-monitoring/node-exporter-5z5ct" Nov 03 01:18:39.365965 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.365908 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/30eede54-1970-4f52-b219-70d3c469ddd4-node-exporter-wtmp\") pod \"node-exporter-5z5ct\" (UID: \"30eede54-1970-4f52-b219-70d3c469ddd4\") " pod="openshift-monitoring/node-exporter-5z5ct" Nov 03 01:18:39.365965 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.365934 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2320cce9-0fdb-470a-ae01-e88118ba0419-crio-socket\") pod \"insights-runtime-extractor-bbfdw\" (UID: \"2320cce9-0fdb-470a-ae01-e88118ba0419\") " pod="openshift-insights/insights-runtime-extractor-bbfdw" Nov 03 01:18:39.366093 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.365968 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/30eede54-1970-4f52-b219-70d3c469ddd4-root\") pod \"node-exporter-5z5ct\" (UID: \"30eede54-1970-4f52-b219-70d3c469ddd4\") " pod="openshift-monitoring/node-exporter-5z5ct" Nov 03 01:18:39.366093 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.365981 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/30eede54-1970-4f52-b219-70d3c469ddd4-sys\") pod \"node-exporter-5z5ct\" (UID: \"30eede54-1970-4f52-b219-70d3c469ddd4\") " pod="openshift-monitoring/node-exporter-5z5ct" Nov 03 01:18:39.366093 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.365995 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/30eede54-1970-4f52-b219-70d3c469ddd4-node-exporter-wtmp\") pod \"node-exporter-5z5ct\" (UID: \"30eede54-1970-4f52-b219-70d3c469ddd4\") " pod="openshift-monitoring/node-exporter-5z5ct" Nov 03 01:18:39.366093 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:39.365992 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2320cce9-0fdb-470a-ae01-e88118ba0419-crio-socket\") pod \"insights-runtime-extractor-bbfdw\" (UID: \"2320cce9-0fdb-470a-ae01-e88118ba0419\") " pod="openshift-insights/insights-runtime-extractor-bbfdw" Nov 03 01:18:40.367781 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:40.367746 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-kmpbz" Nov 03 01:18:41.042899 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:41.042868 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-8bbf59f7f-cmcdf" Nov 03 01:18:42.116670 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:42.116641 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-n2cjg" Nov 03 01:18:42.168024 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:42.167998 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-69fcb6c6b8-jqgxm" Nov 03 01:18:43.126863 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:43.126819 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:43.379288 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:43.379205 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:44.033122 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:18:44.033086 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 01:18:47.299696 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:47.299617 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-kg99x" Nov 03 01:18:48.526109 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:48.526075 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-n2cjg" Nov 03 01:18:48.526522 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:48.526307 5437 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 01:18:48.857677 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:48.857646 5437 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-16.us-west-2.compute.internal" event="NodeReady" Nov 03 01:18:49.405245 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:49.405218 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-ssqxs" Nov 03 01:18:54.953486 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:54.953407 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:55.157967 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:55.157934 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:57.956058 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:57.956013 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:58.203633 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:18:58.203583 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:01.549033 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:19:01.549003 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-dd9jt" Nov 03 01:19:02.358651 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:19:02.358618 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-n2cjg" Nov 03 01:19:07.302312 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:19:07.302284 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-kg99x" Nov 03 01:19:08.359270 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:19:08.359231 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:08.575176 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:19:08.575129 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:11.147962 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:19:11.147924 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:11.391949 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:19:11.391914 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:14.032728 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:19:14.032694 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 01:19:18.358326 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:19:18.358275 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-59d46cbfc-ggljw"] Nov 03 01:19:18.360318 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:19:18.360299 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59d46cbfc-ggljw" Nov 03 01:19:18.417092 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:19:18.417066 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-59d46cbfc-ggljw"] Nov 03 01:19:18.442550 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:19:18.442516 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9ef935c2-5391-4c2d-baaa-9adedbfd47c8-service-ca\") pod \"console-59d46cbfc-ggljw\" (UID: \"9ef935c2-5391-4c2d-baaa-9adedbfd47c8\") " pod="openshift-console/console-59d46cbfc-ggljw" Nov 03 01:19:18.442698 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:19:18.442573 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9ef935c2-5391-4c2d-baaa-9adedbfd47c8-console-serving-cert\") pod \"console-59d46cbfc-ggljw\" (UID: \"9ef935c2-5391-4c2d-baaa-9adedbfd47c8\") " pod="openshift-console/console-59d46cbfc-ggljw" Nov 03 01:19:18.442698 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:19:18.442672 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9ef935c2-5391-4c2d-baaa-9adedbfd47c8-console-oauth-config\") pod \"console-59d46cbfc-ggljw\" (UID: \"9ef935c2-5391-4c2d-baaa-9adedbfd47c8\") " pod="openshift-console/console-59d46cbfc-ggljw" Nov 03 01:19:18.442782 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:19:18.442702 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9ef935c2-5391-4c2d-baaa-9adedbfd47c8-console-config\") pod \"console-59d46cbfc-ggljw\" (UID: \"9ef935c2-5391-4c2d-baaa-9adedbfd47c8\") " pod="openshift-console/console-59d46cbfc-ggljw" Nov 03 01:19:18.442782 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:19:18.442727 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b84t5\" (UniqueName: \"kubernetes.io/projected/9ef935c2-5391-4c2d-baaa-9adedbfd47c8-kube-api-access-b84t5\") pod \"console-59d46cbfc-ggljw\" (UID: \"9ef935c2-5391-4c2d-baaa-9adedbfd47c8\") " pod="openshift-console/console-59d46cbfc-ggljw" Nov 03 01:19:18.442782 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:19:18.442759 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ef935c2-5391-4c2d-baaa-9adedbfd47c8-trusted-ca-bundle\") pod \"console-59d46cbfc-ggljw\" (UID: \"9ef935c2-5391-4c2d-baaa-9adedbfd47c8\") " pod="openshift-console/console-59d46cbfc-ggljw" Nov 03 01:19:18.442876 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:19:18.442848 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9ef935c2-5391-4c2d-baaa-9adedbfd47c8-tmp\") pod \"console-59d46cbfc-ggljw\" (UID: \"9ef935c2-5391-4c2d-baaa-9adedbfd47c8\") " pod="openshift-console/console-59d46cbfc-ggljw" Nov 03 01:19:18.442908 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:19:18.442874 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9ef935c2-5391-4c2d-baaa-9adedbfd47c8-oauth-serving-cert\") pod \"console-59d46cbfc-ggljw\" (UID: \"9ef935c2-5391-4c2d-baaa-9adedbfd47c8\") " pod="openshift-console/console-59d46cbfc-ggljw" Nov 03 01:19:18.543236 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:19:18.543204 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ef935c2-5391-4c2d-baaa-9adedbfd47c8-trusted-ca-bundle\") pod \"console-59d46cbfc-ggljw\" (UID: \"9ef935c2-5391-4c2d-baaa-9adedbfd47c8\") " pod="openshift-console/console-59d46cbfc-ggljw" Nov 03 01:19:18.543437 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:19:18.543247 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9ef935c2-5391-4c2d-baaa-9adedbfd47c8-tmp\") pod \"console-59d46cbfc-ggljw\" (UID: \"9ef935c2-5391-4c2d-baaa-9adedbfd47c8\") " pod="openshift-console/console-59d46cbfc-ggljw" Nov 03 01:19:18.543437 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:19:18.543265 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9ef935c2-5391-4c2d-baaa-9adedbfd47c8-oauth-serving-cert\") pod \"console-59d46cbfc-ggljw\" (UID: \"9ef935c2-5391-4c2d-baaa-9adedbfd47c8\") " pod="openshift-console/console-59d46cbfc-ggljw" Nov 03 01:19:18.543437 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:19:18.543417 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9ef935c2-5391-4c2d-baaa-9adedbfd47c8-service-ca\") pod \"console-59d46cbfc-ggljw\" (UID: \"9ef935c2-5391-4c2d-baaa-9adedbfd47c8\") " pod="openshift-console/console-59d46cbfc-ggljw" Nov 03 01:19:18.543603 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:19:18.543510 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9ef935c2-5391-4c2d-baaa-9adedbfd47c8-console-serving-cert\") pod \"console-59d46cbfc-ggljw\" (UID: \"9ef935c2-5391-4c2d-baaa-9adedbfd47c8\") " pod="openshift-console/console-59d46cbfc-ggljw" Nov 03 01:19:18.543603 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:19:18.543570 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9ef935c2-5391-4c2d-baaa-9adedbfd47c8-console-oauth-config\") pod \"console-59d46cbfc-ggljw\" (UID: \"9ef935c2-5391-4c2d-baaa-9adedbfd47c8\") " pod="openshift-console/console-59d46cbfc-ggljw" Nov 03 01:19:18.543603 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:19:18.543596 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9ef935c2-5391-4c2d-baaa-9adedbfd47c8-console-config\") pod \"console-59d46cbfc-ggljw\" (UID: \"9ef935c2-5391-4c2d-baaa-9adedbfd47c8\") " pod="openshift-console/console-59d46cbfc-ggljw" Nov 03 01:19:18.543757 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:19:18.543632 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b84t5\" (UniqueName: \"kubernetes.io/projected/9ef935c2-5391-4c2d-baaa-9adedbfd47c8-kube-api-access-b84t5\") pod \"console-59d46cbfc-ggljw\" (UID: \"9ef935c2-5391-4c2d-baaa-9adedbfd47c8\") " pod="openshift-console/console-59d46cbfc-ggljw" Nov 03 01:19:18.543757 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:19:18.543714 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9ef935c2-5391-4c2d-baaa-9adedbfd47c8-tmp\") pod \"console-59d46cbfc-ggljw\" (UID: \"9ef935c2-5391-4c2d-baaa-9adedbfd47c8\") " pod="openshift-console/console-59d46cbfc-ggljw" Nov 03 01:19:18.543958 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:19:18.543919 5437 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 03 01:19:18.544071 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:19:18.544015 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9ef935c2-5391-4c2d-baaa-9adedbfd47c8-oauth-serving-cert\") pod \"console-59d46cbfc-ggljw\" (UID: \"9ef935c2-5391-4c2d-baaa-9adedbfd47c8\") " pod="openshift-console/console-59d46cbfc-ggljw" Nov 03 01:19:18.544306 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:19:18.544278 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9ef935c2-5391-4c2d-baaa-9adedbfd47c8-service-ca\") pod \"console-59d46cbfc-ggljw\" (UID: \"9ef935c2-5391-4c2d-baaa-9adedbfd47c8\") " pod="openshift-console/console-59d46cbfc-ggljw" Nov 03 01:19:18.544410 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:19:18.544336 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9ef935c2-5391-4c2d-baaa-9adedbfd47c8-console-config\") pod \"console-59d46cbfc-ggljw\" (UID: \"9ef935c2-5391-4c2d-baaa-9adedbfd47c8\") " pod="openshift-console/console-59d46cbfc-ggljw" Nov 03 01:19:18.544562 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:19:18.544541 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ef935c2-5391-4c2d-baaa-9adedbfd47c8-trusted-ca-bundle\") pod \"console-59d46cbfc-ggljw\" (UID: \"9ef935c2-5391-4c2d-baaa-9adedbfd47c8\") " pod="openshift-console/console-59d46cbfc-ggljw" Nov 03 01:19:18.545994 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:19:18.545969 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9ef935c2-5391-4c2d-baaa-9adedbfd47c8-console-serving-cert\") pod \"console-59d46cbfc-ggljw\" (UID: \"9ef935c2-5391-4c2d-baaa-9adedbfd47c8\") " pod="openshift-console/console-59d46cbfc-ggljw" Nov 03 01:19:18.546108 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:19:18.546087 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9ef935c2-5391-4c2d-baaa-9adedbfd47c8-console-oauth-config\") pod \"console-59d46cbfc-ggljw\" (UID: \"9ef935c2-5391-4c2d-baaa-9adedbfd47c8\") " pod="openshift-console/console-59d46cbfc-ggljw" Nov 03 01:19:18.606649 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:19:18.606617 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b84t5\" (UniqueName: \"kubernetes.io/projected/9ef935c2-5391-4c2d-baaa-9adedbfd47c8-kube-api-access-b84t5\") pod \"console-59d46cbfc-ggljw\" (UID: \"9ef935c2-5391-4c2d-baaa-9adedbfd47c8\") " pod="openshift-console/console-59d46cbfc-ggljw" Nov 03 01:19:18.670641 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:19:18.670560 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59d46cbfc-ggljw" Nov 03 01:19:18.681186 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:19:18.681153 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:18.885428 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:19:18.885392 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:19.081411 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:19:19.081351 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ef935c2_5391_4c2d_baaa_9adedbfd47c8.slice/crio-50fd28a21aab4917c498b17e574caee4e602d3739d47f60e03d4babb67ee9e34 WatchSource:0}: Error finding container 50fd28a21aab4917c498b17e574caee4e602d3739d47f60e03d4babb67ee9e34: Status 404 returned error can't find the container with id 50fd28a21aab4917c498b17e574caee4e602d3739d47f60e03d4babb67ee9e34 Nov 03 01:19:19.082634 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:19:19.082611 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-59d46cbfc-ggljw"] Nov 03 01:19:19.087180 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:19:19.087147 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:19.274748 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:19:19.274713 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59d46cbfc-ggljw" event={"ID":"9ef935c2-5391-4c2d-baaa-9adedbfd47c8","Type":"ContainerStarted","Data":"375d2a4b35bab27f276063c32fb219f37d061215a9eb085657d65220f4f77140"} Nov 03 01:19:19.274907 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:19:19.274759 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59d46cbfc-ggljw" event={"ID":"9ef935c2-5391-4c2d-baaa-9adedbfd47c8","Type":"ContainerStarted","Data":"50fd28a21aab4917c498b17e574caee4e602d3739d47f60e03d4babb67ee9e34"} Nov 03 01:19:19.395477 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:19:19.395385 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-59d46cbfc-ggljw" podStartSLOduration=1.395352302 podStartE2EDuration="1.395352302s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:19:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:19:19.395330406 +0000 UTC m=+41.424812713" watchObservedRunningTime="2026-11-03 01:19:19.395352302 +0000 UTC m=+41.424834593" Nov 03 01:19:28.671099 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:19:28.671063 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-59d46cbfc-ggljw" Nov 03 01:19:28.671567 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:19:28.671127 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-59d46cbfc-ggljw" Nov 03 01:19:28.675959 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:19:28.675936 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-59d46cbfc-ggljw" Nov 03 01:19:29.310797 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:19:29.310768 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-59d46cbfc-ggljw" Nov 03 01:19:29.485060 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:19:29.485022 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-69fcb6c6b8-jqgxm"] Nov 03 01:19:31.265509 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:19:31.265465 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:35.851763 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:19:35.851723 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:44.032650 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:19:44.032612 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 01:19:54.504044 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:19:54.503986 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-69fcb6c6b8-jqgxm" podUID="386397a9-5c8d-4221-894b-279e0b396afe" containerName="console" containerID="cri-o://9a195ef36f598769559baef0a3d59ff9ce64c2296d588760a3562463e5240eb3" gracePeriod=15 Nov 03 01:19:54.526283 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:19:54.526252 5437 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod386397a9_5c8d_4221_894b_279e0b396afe.slice/crio-9a195ef36f598769559baef0a3d59ff9ce64c2296d588760a3562463e5240eb3.scope\": RecentStats: unable to find data in memory cache]" Nov 03 01:19:54.743246 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:19:54.743219 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69fcb6c6b8-jqgxm" Nov 03 01:19:54.820861 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:19:54.820825 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/386397a9-5c8d-4221-894b-279e0b396afe-console-config\" (UniqueName: \"kubernetes.io/configmap/386397a9-5c8d-4221-894b-279e0b396afe-console-config\") pod \"386397a9-5c8d-4221-894b-279e0b396afe\" (UID: \"386397a9-5c8d-4221-894b-279e0b396afe\") " Nov 03 01:19:54.821058 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:19:54.820882 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/386397a9-5c8d-4221-894b-279e0b396afe-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/386397a9-5c8d-4221-894b-279e0b396afe-oauth-serving-cert\") pod \"386397a9-5c8d-4221-894b-279e0b396afe\" (UID: \"386397a9-5c8d-4221-894b-279e0b396afe\") " Nov 03 01:19:54.821058 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:19:54.820927 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/386397a9-5c8d-4221-894b-279e0b396afe-tmp\" (UniqueName: \"kubernetes.io/empty-dir/386397a9-5c8d-4221-894b-279e0b396afe-tmp\") pod \"386397a9-5c8d-4221-894b-279e0b396afe\" (UID: \"386397a9-5c8d-4221-894b-279e0b396afe\") " Nov 03 01:19:54.821058 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:19:54.820963 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/386397a9-5c8d-4221-894b-279e0b396afe-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/386397a9-5c8d-4221-894b-279e0b396afe-trusted-ca-bundle\") pod \"386397a9-5c8d-4221-894b-279e0b396afe\" (UID: \"386397a9-5c8d-4221-894b-279e0b396afe\") " Nov 03 01:19:54.821058 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:19:54.820986 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/386397a9-5c8d-4221-894b-279e0b396afe-service-ca\" (UniqueName: \"kubernetes.io/configmap/386397a9-5c8d-4221-894b-279e0b396afe-service-ca\") pod \"386397a9-5c8d-4221-894b-279e0b396afe\" (UID: \"386397a9-5c8d-4221-894b-279e0b396afe\") " Nov 03 01:19:54.821058 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:19:54.821028 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/386397a9-5c8d-4221-894b-279e0b396afe-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/386397a9-5c8d-4221-894b-279e0b396afe-console-oauth-config\") pod \"386397a9-5c8d-4221-894b-279e0b396afe\" (UID: \"386397a9-5c8d-4221-894b-279e0b396afe\") " Nov 03 01:19:54.821308 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:19:54.821075 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/386397a9-5c8d-4221-894b-279e0b396afe-kube-api-access-wk6dw\" (UniqueName: \"kubernetes.io/projected/386397a9-5c8d-4221-894b-279e0b396afe-kube-api-access-wk6dw\") pod \"386397a9-5c8d-4221-894b-279e0b396afe\" (UID: \"386397a9-5c8d-4221-894b-279e0b396afe\") " Nov 03 01:19:54.821308 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:19:54.821187 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/386397a9-5c8d-4221-894b-279e0b396afe-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/386397a9-5c8d-4221-894b-279e0b396afe-console-serving-cert\") pod \"386397a9-5c8d-4221-894b-279e0b396afe\" (UID: \"386397a9-5c8d-4221-894b-279e0b396afe\") " Nov 03 01:19:54.821308 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:19:54.821276 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/386397a9-5c8d-4221-894b-279e0b396afe-console-config" pod "386397a9-5c8d-4221-894b-279e0b396afe" (UID: "386397a9-5c8d-4221-894b-279e0b396afe"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:19:54.821517 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:19:54.821336 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/386397a9-5c8d-4221-894b-279e0b396afe-oauth-serving-cert" pod "386397a9-5c8d-4221-894b-279e0b396afe" (UID: "386397a9-5c8d-4221-894b-279e0b396afe"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:19:54.821517 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:19:54.821444 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/386397a9-5c8d-4221-894b-279e0b396afe-tmp" pod "386397a9-5c8d-4221-894b-279e0b396afe" (UID: "386397a9-5c8d-4221-894b-279e0b396afe"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:19:54.821618 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:19:54.821576 5437 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/386397a9-5c8d-4221-894b-279e0b396afe-tmp\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:19:54.821618 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:19:54.821592 5437 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/386397a9-5c8d-4221-894b-279e0b396afe-console-config\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:19:54.821618 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:19:54.821602 5437 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/386397a9-5c8d-4221-894b-279e0b396afe-oauth-serving-cert\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:19:54.821764 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:19:54.821646 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/386397a9-5c8d-4221-894b-279e0b396afe-service-ca" pod "386397a9-5c8d-4221-894b-279e0b396afe" (UID: "386397a9-5c8d-4221-894b-279e0b396afe"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:19:54.821764 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:19:54.821658 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/386397a9-5c8d-4221-894b-279e0b396afe-trusted-ca-bundle" pod "386397a9-5c8d-4221-894b-279e0b396afe" (UID: "386397a9-5c8d-4221-894b-279e0b396afe"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:19:54.823403 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:19:54.823374 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/386397a9-5c8d-4221-894b-279e0b396afe-kube-api-access-wk6dw" pod "386397a9-5c8d-4221-894b-279e0b396afe" (UID: "386397a9-5c8d-4221-894b-279e0b396afe"). InnerVolumeSpecName "kube-api-access-wk6dw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:19:54.823403 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:19:54.823377 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/386397a9-5c8d-4221-894b-279e0b396afe-console-oauth-config" pod "386397a9-5c8d-4221-894b-279e0b396afe" (UID: "386397a9-5c8d-4221-894b-279e0b396afe"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:19:54.823615 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:19:54.823486 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/386397a9-5c8d-4221-894b-279e0b396afe-console-serving-cert" pod "386397a9-5c8d-4221-894b-279e0b396afe" (UID: "386397a9-5c8d-4221-894b-279e0b396afe"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:19:54.922891 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:19:54.922858 5437 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/386397a9-5c8d-4221-894b-279e0b396afe-console-serving-cert\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:19:54.922891 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:19:54.922884 5437 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/386397a9-5c8d-4221-894b-279e0b396afe-trusted-ca-bundle\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:19:54.922891 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:19:54.922894 5437 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/386397a9-5c8d-4221-894b-279e0b396afe-service-ca\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:19:54.923113 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:19:54.922903 5437 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/386397a9-5c8d-4221-894b-279e0b396afe-console-oauth-config\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:19:54.923113 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:19:54.922912 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wk6dw\" (UniqueName: \"kubernetes.io/projected/386397a9-5c8d-4221-894b-279e0b396afe-kube-api-access-wk6dw\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:19:55.391894 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:19:55.391864 5437 generic.go:356] "Generic (PLEG): container finished" podID="386397a9-5c8d-4221-894b-279e0b396afe" containerID="9a195ef36f598769559baef0a3d59ff9ce64c2296d588760a3562463e5240eb3" exitCode=0 Nov 03 01:19:55.392068 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:19:55.391911 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69fcb6c6b8-jqgxm" Nov 03 01:19:55.392068 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:19:55.391935 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69fcb6c6b8-jqgxm" event={"ID":"386397a9-5c8d-4221-894b-279e0b396afe","Type":"ContainerDied","Data":"9a195ef36f598769559baef0a3d59ff9ce64c2296d588760a3562463e5240eb3"} Nov 03 01:19:55.392068 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:19:55.391973 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69fcb6c6b8-jqgxm" event={"ID":"386397a9-5c8d-4221-894b-279e0b396afe","Type":"ContainerDied","Data":"7e4535033e2187d3a887d160f43211e0b3f012c6b23c0b04d27c966b04e03c50"} Nov 03 01:19:55.392068 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:19:55.391992 5437 scope.go:122] "RemoveContainer" containerID="9a195ef36f598769559baef0a3d59ff9ce64c2296d588760a3562463e5240eb3" Nov 03 01:19:55.400501 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:19:55.400449 5437 scope.go:122] "RemoveContainer" containerID="9a195ef36f598769559baef0a3d59ff9ce64c2296d588760a3562463e5240eb3" Nov 03 01:19:55.400774 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:19:55.400757 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a195ef36f598769559baef0a3d59ff9ce64c2296d588760a3562463e5240eb3\": container with ID starting with 9a195ef36f598769559baef0a3d59ff9ce64c2296d588760a3562463e5240eb3 not found: ID does not exist" containerID="9a195ef36f598769559baef0a3d59ff9ce64c2296d588760a3562463e5240eb3" Nov 03 01:19:55.400816 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:19:55.400784 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a195ef36f598769559baef0a3d59ff9ce64c2296d588760a3562463e5240eb3"} err="failed to get container status \"9a195ef36f598769559baef0a3d59ff9ce64c2296d588760a3562463e5240eb3\": rpc error: code = NotFound desc = could not find container \"9a195ef36f598769559baef0a3d59ff9ce64c2296d588760a3562463e5240eb3\": container with ID starting with 9a195ef36f598769559baef0a3d59ff9ce64c2296d588760a3562463e5240eb3 not found: ID does not exist" Nov 03 01:19:55.561181 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:19:55.561144 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-69fcb6c6b8-jqgxm"] Nov 03 01:19:55.613006 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:19:55.612972 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-69fcb6c6b8-jqgxm"] Nov 03 01:19:56.085585 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:19:56.085544 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="386397a9-5c8d-4221-894b-279e0b396afe" path="/var/lib/kubelet/pods/386397a9-5c8d-4221-894b-279e0b396afe/volumes" Nov 03 01:20:06.305702 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:20:06.305660 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:06.507092 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:20:06.507058 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:14.033828 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:20:14.033793 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 01:20:18.865235 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:20:18.865156 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:19.125748 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:20:19.125659 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:23.604564 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:20:23.604522 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:27.956327 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:20:27.956295 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:28.161568 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:20:28.161529 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:29.051902 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:20:29.051868 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-55c8dc5cf6-bfx9c"] Nov 03 01:20:29.052282 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:20:29.052143 5437 cpu_manager.go:414] "containerMap: removing container" podUID="386397a9-5c8d-4221-894b-279e0b396afe" containerName="console" Nov 03 01:20:29.052282 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:20:29.052154 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="386397a9-5c8d-4221-894b-279e0b396afe" containerName="console" Nov 03 01:20:29.052282 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:20:29.052220 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="386397a9-5c8d-4221-894b-279e0b396afe" containerName="console" Nov 03 01:20:29.054030 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:20:29.054014 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55c8dc5cf6-bfx9c" Nov 03 01:20:29.160394 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:20:29.160338 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-55c8dc5cf6-bfx9c"] Nov 03 01:20:29.183011 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:20:29.182979 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rw5g7\" (UniqueName: \"kubernetes.io/projected/6e3a6316-1c6c-4bb7-a110-18adc6ca4f80-kube-api-access-rw5g7\") pod \"console-55c8dc5cf6-bfx9c\" (UID: \"6e3a6316-1c6c-4bb7-a110-18adc6ca4f80\") " pod="openshift-console/console-55c8dc5cf6-bfx9c" Nov 03 01:20:29.183152 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:20:29.183024 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6e3a6316-1c6c-4bb7-a110-18adc6ca4f80-service-ca\") pod \"console-55c8dc5cf6-bfx9c\" (UID: \"6e3a6316-1c6c-4bb7-a110-18adc6ca4f80\") " pod="openshift-console/console-55c8dc5cf6-bfx9c" Nov 03 01:20:29.183152 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:20:29.183043 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6e3a6316-1c6c-4bb7-a110-18adc6ca4f80-oauth-serving-cert\") pod \"console-55c8dc5cf6-bfx9c\" (UID: \"6e3a6316-1c6c-4bb7-a110-18adc6ca4f80\") " pod="openshift-console/console-55c8dc5cf6-bfx9c" Nov 03 01:20:29.183224 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:20:29.183141 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6e3a6316-1c6c-4bb7-a110-18adc6ca4f80-tmp\") pod \"console-55c8dc5cf6-bfx9c\" (UID: \"6e3a6316-1c6c-4bb7-a110-18adc6ca4f80\") " pod="openshift-console/console-55c8dc5cf6-bfx9c" Nov 03 01:20:29.183224 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:20:29.183178 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6e3a6316-1c6c-4bb7-a110-18adc6ca4f80-console-config\") pod \"console-55c8dc5cf6-bfx9c\" (UID: \"6e3a6316-1c6c-4bb7-a110-18adc6ca4f80\") " pod="openshift-console/console-55c8dc5cf6-bfx9c" Nov 03 01:20:29.183224 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:20:29.183208 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e3a6316-1c6c-4bb7-a110-18adc6ca4f80-trusted-ca-bundle\") pod \"console-55c8dc5cf6-bfx9c\" (UID: \"6e3a6316-1c6c-4bb7-a110-18adc6ca4f80\") " pod="openshift-console/console-55c8dc5cf6-bfx9c" Nov 03 01:20:29.183320 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:20:29.183286 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6e3a6316-1c6c-4bb7-a110-18adc6ca4f80-console-oauth-config\") pod \"console-55c8dc5cf6-bfx9c\" (UID: \"6e3a6316-1c6c-4bb7-a110-18adc6ca4f80\") " pod="openshift-console/console-55c8dc5cf6-bfx9c" Nov 03 01:20:29.183356 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:20:29.183317 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6e3a6316-1c6c-4bb7-a110-18adc6ca4f80-console-serving-cert\") pod \"console-55c8dc5cf6-bfx9c\" (UID: \"6e3a6316-1c6c-4bb7-a110-18adc6ca4f80\") " pod="openshift-console/console-55c8dc5cf6-bfx9c" Nov 03 01:20:29.284455 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:20:29.284416 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e3a6316-1c6c-4bb7-a110-18adc6ca4f80-trusted-ca-bundle\") pod \"console-55c8dc5cf6-bfx9c\" (UID: \"6e3a6316-1c6c-4bb7-a110-18adc6ca4f80\") " pod="openshift-console/console-55c8dc5cf6-bfx9c" Nov 03 01:20:29.284455 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:20:29.284453 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6e3a6316-1c6c-4bb7-a110-18adc6ca4f80-console-oauth-config\") pod \"console-55c8dc5cf6-bfx9c\" (UID: \"6e3a6316-1c6c-4bb7-a110-18adc6ca4f80\") " pod="openshift-console/console-55c8dc5cf6-bfx9c" Nov 03 01:20:29.284718 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:20:29.284476 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6e3a6316-1c6c-4bb7-a110-18adc6ca4f80-console-serving-cert\") pod \"console-55c8dc5cf6-bfx9c\" (UID: \"6e3a6316-1c6c-4bb7-a110-18adc6ca4f80\") " pod="openshift-console/console-55c8dc5cf6-bfx9c" Nov 03 01:20:29.284718 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:20:29.284501 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rw5g7\" (UniqueName: \"kubernetes.io/projected/6e3a6316-1c6c-4bb7-a110-18adc6ca4f80-kube-api-access-rw5g7\") pod \"console-55c8dc5cf6-bfx9c\" (UID: \"6e3a6316-1c6c-4bb7-a110-18adc6ca4f80\") " pod="openshift-console/console-55c8dc5cf6-bfx9c" Nov 03 01:20:29.284718 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:20:29.284528 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6e3a6316-1c6c-4bb7-a110-18adc6ca4f80-service-ca\") pod \"console-55c8dc5cf6-bfx9c\" (UID: \"6e3a6316-1c6c-4bb7-a110-18adc6ca4f80\") " pod="openshift-console/console-55c8dc5cf6-bfx9c" Nov 03 01:20:29.284718 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:20:29.284544 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6e3a6316-1c6c-4bb7-a110-18adc6ca4f80-oauth-serving-cert\") pod \"console-55c8dc5cf6-bfx9c\" (UID: \"6e3a6316-1c6c-4bb7-a110-18adc6ca4f80\") " pod="openshift-console/console-55c8dc5cf6-bfx9c" Nov 03 01:20:29.284718 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:20:29.284567 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6e3a6316-1c6c-4bb7-a110-18adc6ca4f80-tmp\") pod \"console-55c8dc5cf6-bfx9c\" (UID: \"6e3a6316-1c6c-4bb7-a110-18adc6ca4f80\") " pod="openshift-console/console-55c8dc5cf6-bfx9c" Nov 03 01:20:29.284718 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:20:29.284585 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6e3a6316-1c6c-4bb7-a110-18adc6ca4f80-console-config\") pod \"console-55c8dc5cf6-bfx9c\" (UID: \"6e3a6316-1c6c-4bb7-a110-18adc6ca4f80\") " pod="openshift-console/console-55c8dc5cf6-bfx9c" Nov 03 01:20:29.285255 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:20:29.285225 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6e3a6316-1c6c-4bb7-a110-18adc6ca4f80-tmp\") pod \"console-55c8dc5cf6-bfx9c\" (UID: \"6e3a6316-1c6c-4bb7-a110-18adc6ca4f80\") " pod="openshift-console/console-55c8dc5cf6-bfx9c" Nov 03 01:20:29.285517 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:20:29.285496 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6e3a6316-1c6c-4bb7-a110-18adc6ca4f80-service-ca\") pod \"console-55c8dc5cf6-bfx9c\" (UID: \"6e3a6316-1c6c-4bb7-a110-18adc6ca4f80\") " pod="openshift-console/console-55c8dc5cf6-bfx9c" Nov 03 01:20:29.285602 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:20:29.285511 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6e3a6316-1c6c-4bb7-a110-18adc6ca4f80-oauth-serving-cert\") pod \"console-55c8dc5cf6-bfx9c\" (UID: \"6e3a6316-1c6c-4bb7-a110-18adc6ca4f80\") " pod="openshift-console/console-55c8dc5cf6-bfx9c" Nov 03 01:20:29.285602 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:20:29.285531 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6e3a6316-1c6c-4bb7-a110-18adc6ca4f80-console-config\") pod \"console-55c8dc5cf6-bfx9c\" (UID: \"6e3a6316-1c6c-4bb7-a110-18adc6ca4f80\") " pod="openshift-console/console-55c8dc5cf6-bfx9c" Nov 03 01:20:29.285602 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:20:29.285586 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e3a6316-1c6c-4bb7-a110-18adc6ca4f80-trusted-ca-bundle\") pod \"console-55c8dc5cf6-bfx9c\" (UID: \"6e3a6316-1c6c-4bb7-a110-18adc6ca4f80\") " pod="openshift-console/console-55c8dc5cf6-bfx9c" Nov 03 01:20:29.287128 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:20:29.287100 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6e3a6316-1c6c-4bb7-a110-18adc6ca4f80-console-oauth-config\") pod \"console-55c8dc5cf6-bfx9c\" (UID: \"6e3a6316-1c6c-4bb7-a110-18adc6ca4f80\") " pod="openshift-console/console-55c8dc5cf6-bfx9c" Nov 03 01:20:29.287216 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:20:29.287109 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6e3a6316-1c6c-4bb7-a110-18adc6ca4f80-console-serving-cert\") pod \"console-55c8dc5cf6-bfx9c\" (UID: \"6e3a6316-1c6c-4bb7-a110-18adc6ca4f80\") " pod="openshift-console/console-55c8dc5cf6-bfx9c" Nov 03 01:20:29.347910 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:20:29.347832 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rw5g7\" (UniqueName: \"kubernetes.io/projected/6e3a6316-1c6c-4bb7-a110-18adc6ca4f80-kube-api-access-rw5g7\") pod \"console-55c8dc5cf6-bfx9c\" (UID: \"6e3a6316-1c6c-4bb7-a110-18adc6ca4f80\") " pod="openshift-console/console-55c8dc5cf6-bfx9c" Nov 03 01:20:29.363203 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:20:29.363168 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55c8dc5cf6-bfx9c" Nov 03 01:20:29.371132 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:20:29.371103 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:29.575430 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:20:29.575391 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:29.729129 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:20:29.729093 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e3a6316_1c6c_4bb7_a110_18adc6ca4f80.slice/crio-e8aa866818cd49a9c209fcdc76f82099bdbf2385e053b226e6ae9d109b279723 WatchSource:0}: Error finding container e8aa866818cd49a9c209fcdc76f82099bdbf2385e053b226e6ae9d109b279723: Status 404 returned error can't find the container with id e8aa866818cd49a9c209fcdc76f82099bdbf2385e053b226e6ae9d109b279723 Nov 03 01:20:29.730508 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:20:29.730483 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-55c8dc5cf6-bfx9c"] Nov 03 01:20:29.777991 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:20:29.777951 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:30.510247 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:20:30.510210 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55c8dc5cf6-bfx9c" event={"ID":"6e3a6316-1c6c-4bb7-a110-18adc6ca4f80","Type":"ContainerStarted","Data":"2026330953e23f7ba43215220fbdf8b46532d8ae3991cd2818c94daea4190511"} Nov 03 01:20:30.510684 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:20:30.510267 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55c8dc5cf6-bfx9c" event={"ID":"6e3a6316-1c6c-4bb7-a110-18adc6ca4f80","Type":"ContainerStarted","Data":"e8aa866818cd49a9c209fcdc76f82099bdbf2385e053b226e6ae9d109b279723"} Nov 03 01:20:30.683807 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:20:30.683746 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-55c8dc5cf6-bfx9c" podStartSLOduration=1.683731313 podStartE2EDuration="1.683731313s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:20:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:20:30.631902315 +0000 UTC m=+112.661384609" watchObservedRunningTime="2026-11-03 01:20:30.683731313 +0000 UTC m=+112.713213608" Nov 03 01:20:32.324904 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:20:32.324866 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:36.191775 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:20:36.191739 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:39.363786 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:20:39.363750 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-55c8dc5cf6-bfx9c" Nov 03 01:20:39.364208 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:20:39.364045 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-55c8dc5cf6-bfx9c" Nov 03 01:20:39.368597 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:20:39.368576 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-55c8dc5cf6-bfx9c" Nov 03 01:20:39.542949 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:20:39.542917 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-55c8dc5cf6-bfx9c" Nov 03 01:20:39.693395 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:20:39.693293 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-59d46cbfc-ggljw"] Nov 03 01:20:44.032535 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:20:44.032489 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 01:21:04.713819 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:21:04.713756 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-59d46cbfc-ggljw" podUID="9ef935c2-5391-4c2d-baaa-9adedbfd47c8" containerName="console" containerID="cri-o://375d2a4b35bab27f276063c32fb219f37d061215a9eb085657d65220f4f77140" gracePeriod=15 Nov 03 01:21:04.958244 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:21:04.958215 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59d46cbfc-ggljw" Nov 03 01:21:05.050258 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:21:05.050219 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ef935c2-5391-4c2d-baaa-9adedbfd47c8-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9ef935c2-5391-4c2d-baaa-9adedbfd47c8-oauth-serving-cert\") pod \"9ef935c2-5391-4c2d-baaa-9adedbfd47c8\" (UID: \"9ef935c2-5391-4c2d-baaa-9adedbfd47c8\") " Nov 03 01:21:05.050424 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:21:05.050276 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ef935c2-5391-4c2d-baaa-9adedbfd47c8-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ef935c2-5391-4c2d-baaa-9adedbfd47c8-service-ca\") pod \"9ef935c2-5391-4c2d-baaa-9adedbfd47c8\" (UID: \"9ef935c2-5391-4c2d-baaa-9adedbfd47c8\") " Nov 03 01:21:05.050424 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:21:05.050314 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9ef935c2-5391-4c2d-baaa-9adedbfd47c8-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9ef935c2-5391-4c2d-baaa-9adedbfd47c8-console-oauth-config\") pod \"9ef935c2-5391-4c2d-baaa-9adedbfd47c8\" (UID: \"9ef935c2-5391-4c2d-baaa-9adedbfd47c8\") " Nov 03 01:21:05.050424 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:21:05.050339 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ef935c2-5391-4c2d-baaa-9adedbfd47c8-console-config\" (UniqueName: \"kubernetes.io/configmap/9ef935c2-5391-4c2d-baaa-9adedbfd47c8-console-config\") pod \"9ef935c2-5391-4c2d-baaa-9adedbfd47c8\" (UID: \"9ef935c2-5391-4c2d-baaa-9adedbfd47c8\") " Nov 03 01:21:05.050577 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:21:05.050421 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ef935c2-5391-4c2d-baaa-9adedbfd47c8-tmp\" (UniqueName: \"kubernetes.io/empty-dir/9ef935c2-5391-4c2d-baaa-9adedbfd47c8-tmp\") pod \"9ef935c2-5391-4c2d-baaa-9adedbfd47c8\" (UID: \"9ef935c2-5391-4c2d-baaa-9adedbfd47c8\") " Nov 03 01:21:05.050577 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:21:05.050489 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ef935c2-5391-4c2d-baaa-9adedbfd47c8-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ef935c2-5391-4c2d-baaa-9adedbfd47c8-trusted-ca-bundle\") pod \"9ef935c2-5391-4c2d-baaa-9adedbfd47c8\" (UID: \"9ef935c2-5391-4c2d-baaa-9adedbfd47c8\") " Nov 03 01:21:05.050577 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:21:05.050562 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9ef935c2-5391-4c2d-baaa-9adedbfd47c8-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9ef935c2-5391-4c2d-baaa-9adedbfd47c8-console-serving-cert\") pod \"9ef935c2-5391-4c2d-baaa-9adedbfd47c8\" (UID: \"9ef935c2-5391-4c2d-baaa-9adedbfd47c8\") " Nov 03 01:21:05.050724 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:21:05.050604 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ef935c2-5391-4c2d-baaa-9adedbfd47c8-kube-api-access-b84t5\" (UniqueName: \"kubernetes.io/projected/9ef935c2-5391-4c2d-baaa-9adedbfd47c8-kube-api-access-b84t5\") pod \"9ef935c2-5391-4c2d-baaa-9adedbfd47c8\" (UID: \"9ef935c2-5391-4c2d-baaa-9adedbfd47c8\") " Nov 03 01:21:05.050724 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:21:05.050696 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ef935c2-5391-4c2d-baaa-9adedbfd47c8-oauth-serving-cert" pod "9ef935c2-5391-4c2d-baaa-9adedbfd47c8" (UID: "9ef935c2-5391-4c2d-baaa-9adedbfd47c8"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:21:05.050870 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:21:05.050844 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ef935c2-5391-4c2d-baaa-9adedbfd47c8-service-ca" pod "9ef935c2-5391-4c2d-baaa-9adedbfd47c8" (UID: "9ef935c2-5391-4c2d-baaa-9adedbfd47c8"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:21:05.050927 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:21:05.050895 5437 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9ef935c2-5391-4c2d-baaa-9adedbfd47c8-oauth-serving-cert\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:21:05.051031 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:21:05.050977 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ef935c2-5391-4c2d-baaa-9adedbfd47c8-tmp" pod "9ef935c2-5391-4c2d-baaa-9adedbfd47c8" (UID: "9ef935c2-5391-4c2d-baaa-9adedbfd47c8"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:21:05.051031 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:21:05.050873 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ef935c2-5391-4c2d-baaa-9adedbfd47c8-console-config" pod "9ef935c2-5391-4c2d-baaa-9adedbfd47c8" (UID: "9ef935c2-5391-4c2d-baaa-9adedbfd47c8"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:21:05.051220 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:21:05.051093 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ef935c2-5391-4c2d-baaa-9adedbfd47c8-trusted-ca-bundle" pod "9ef935c2-5391-4c2d-baaa-9adedbfd47c8" (UID: "9ef935c2-5391-4c2d-baaa-9adedbfd47c8"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:21:05.052646 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:21:05.052621 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ef935c2-5391-4c2d-baaa-9adedbfd47c8-console-oauth-config" pod "9ef935c2-5391-4c2d-baaa-9adedbfd47c8" (UID: "9ef935c2-5391-4c2d-baaa-9adedbfd47c8"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:21:05.052840 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:21:05.052820 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ef935c2-5391-4c2d-baaa-9adedbfd47c8-kube-api-access-b84t5" pod "9ef935c2-5391-4c2d-baaa-9adedbfd47c8" (UID: "9ef935c2-5391-4c2d-baaa-9adedbfd47c8"). InnerVolumeSpecName "kube-api-access-b84t5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:21:05.052901 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:21:05.052845 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ef935c2-5391-4c2d-baaa-9adedbfd47c8-console-serving-cert" pod "9ef935c2-5391-4c2d-baaa-9adedbfd47c8" (UID: "9ef935c2-5391-4c2d-baaa-9adedbfd47c8"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:21:05.152146 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:21:05.152091 5437 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ef935c2-5391-4c2d-baaa-9adedbfd47c8-trusted-ca-bundle\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:21:05.152146 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:21:05.152124 5437 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9ef935c2-5391-4c2d-baaa-9adedbfd47c8-console-serving-cert\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:21:05.152146 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:21:05.152134 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b84t5\" (UniqueName: \"kubernetes.io/projected/9ef935c2-5391-4c2d-baaa-9adedbfd47c8-kube-api-access-b84t5\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:21:05.152146 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:21:05.152145 5437 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9ef935c2-5391-4c2d-baaa-9adedbfd47c8-service-ca\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:21:05.152146 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:21:05.152155 5437 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9ef935c2-5391-4c2d-baaa-9adedbfd47c8-console-oauth-config\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:21:05.152484 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:21:05.152234 5437 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9ef935c2-5391-4c2d-baaa-9adedbfd47c8-console-config\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:21:05.152484 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:21:05.152244 5437 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9ef935c2-5391-4c2d-baaa-9adedbfd47c8-tmp\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:21:05.626939 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:21:05.626901 5437 generic.go:356] "Generic (PLEG): container finished" podID="9ef935c2-5391-4c2d-baaa-9adedbfd47c8" containerID="375d2a4b35bab27f276063c32fb219f37d061215a9eb085657d65220f4f77140" exitCode=0 Nov 03 01:21:05.627112 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:21:05.626950 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59d46cbfc-ggljw" Nov 03 01:21:05.627112 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:21:05.626970 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59d46cbfc-ggljw" event={"ID":"9ef935c2-5391-4c2d-baaa-9adedbfd47c8","Type":"ContainerDied","Data":"375d2a4b35bab27f276063c32fb219f37d061215a9eb085657d65220f4f77140"} Nov 03 01:21:05.627112 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:21:05.627015 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59d46cbfc-ggljw" event={"ID":"9ef935c2-5391-4c2d-baaa-9adedbfd47c8","Type":"ContainerDied","Data":"50fd28a21aab4917c498b17e574caee4e602d3739d47f60e03d4babb67ee9e34"} Nov 03 01:21:05.627112 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:21:05.627038 5437 scope.go:122] "RemoveContainer" containerID="375d2a4b35bab27f276063c32fb219f37d061215a9eb085657d65220f4f77140" Nov 03 01:21:05.635653 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:21:05.635633 5437 scope.go:122] "RemoveContainer" containerID="375d2a4b35bab27f276063c32fb219f37d061215a9eb085657d65220f4f77140" Nov 03 01:21:05.635981 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:21:05.635962 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"375d2a4b35bab27f276063c32fb219f37d061215a9eb085657d65220f4f77140\": container with ID starting with 375d2a4b35bab27f276063c32fb219f37d061215a9eb085657d65220f4f77140 not found: ID does not exist" containerID="375d2a4b35bab27f276063c32fb219f37d061215a9eb085657d65220f4f77140" Nov 03 01:21:05.636029 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:21:05.635991 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"375d2a4b35bab27f276063c32fb219f37d061215a9eb085657d65220f4f77140"} err="failed to get container status \"375d2a4b35bab27f276063c32fb219f37d061215a9eb085657d65220f4f77140\": rpc error: code = NotFound desc = could not find container \"375d2a4b35bab27f276063c32fb219f37d061215a9eb085657d65220f4f77140\": container with ID starting with 375d2a4b35bab27f276063c32fb219f37d061215a9eb085657d65220f4f77140 not found: ID does not exist" Nov 03 01:21:05.796200 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:21:05.796168 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-59d46cbfc-ggljw"] Nov 03 01:21:05.848143 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:21:05.848101 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-59d46cbfc-ggljw"] Nov 03 01:21:06.085414 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:21:06.085351 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ef935c2-5391-4c2d-baaa-9adedbfd47c8" path="/var/lib/kubelet/pods/9ef935c2-5391-4c2d-baaa-9adedbfd47c8/volumes" Nov 03 01:21:14.033455 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:21:14.033420 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 01:21:18.477566 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:21:18.477534 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:18.730968 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:21:18.730880 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:22.409495 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:21:22.409457 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:22.671519 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:21:22.671428 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:36.548246 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:21:36.548206 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:44.032967 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:21:44.032926 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 01:22:14.032997 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:22:14.032914 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 01:22:19.479221 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:22:19.479185 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:22:20.306191 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:22:20.306153 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7r5fc"] Nov 03 01:22:20.306549 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:22:20.306533 5437 cpu_manager.go:414] "containerMap: removing container" podUID="9ef935c2-5391-4c2d-baaa-9adedbfd47c8" containerName="console" Nov 03 01:22:20.306623 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:22:20.306551 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ef935c2-5391-4c2d-baaa-9adedbfd47c8" containerName="console" Nov 03 01:22:20.306680 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:22:20.306654 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ef935c2-5391-4c2d-baaa-9adedbfd47c8" containerName="console" Nov 03 01:22:20.309056 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:22:20.309038 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7r5fc" Nov 03 01:22:20.311570 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:22:20.311546 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vrz4g"] Nov 03 01:22:20.313732 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:22:20.313714 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vrz4g" Nov 03 01:22:20.367348 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:22:20.367315 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Nov 03 01:22:20.367348 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:22:20.367345 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Nov 03 01:22:20.367686 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:22:20.367671 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-4lhp2\"" Nov 03 01:22:20.409864 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:22:20.409833 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a8e50012-20af-48e8-8d8e-9909a6e9dce1-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vrz4g\" (UID: \"a8e50012-20af-48e8-8d8e-9909a6e9dce1\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vrz4g" Nov 03 01:22:20.410064 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:22:20.409885 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dtvg\" (UniqueName: \"kubernetes.io/projected/a8e50012-20af-48e8-8d8e-9909a6e9dce1-kube-api-access-9dtvg\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vrz4g\" (UID: \"a8e50012-20af-48e8-8d8e-9909a6e9dce1\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vrz4g" Nov 03 01:22:20.410064 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:22:20.409918 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnzgs\" (UniqueName: \"kubernetes.io/projected/3ff66d4e-8a7c-4191-8334-03618d2712a2-kube-api-access-jnzgs\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7r5fc\" (UID: \"3ff66d4e-8a7c-4191-8334-03618d2712a2\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7r5fc" Nov 03 01:22:20.410064 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:22:20.410022 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a8e50012-20af-48e8-8d8e-9909a6e9dce1-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vrz4g\" (UID: \"a8e50012-20af-48e8-8d8e-9909a6e9dce1\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vrz4g" Nov 03 01:22:20.410251 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:22:20.410078 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3ff66d4e-8a7c-4191-8334-03618d2712a2-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7r5fc\" (UID: \"3ff66d4e-8a7c-4191-8334-03618d2712a2\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7r5fc" Nov 03 01:22:20.410251 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:22:20.410161 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3ff66d4e-8a7c-4191-8334-03618d2712a2-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7r5fc\" (UID: \"3ff66d4e-8a7c-4191-8334-03618d2712a2\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7r5fc" Nov 03 01:22:20.419168 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:22:20.419147 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-wwcg5\"" Nov 03 01:22:20.419691 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:22:20.419674 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7r5fc"] Nov 03 01:22:20.419961 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:22:20.419944 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vrz4g"] Nov 03 01:22:20.511518 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:22:20.511487 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dtvg\" (UniqueName: \"kubernetes.io/projected/a8e50012-20af-48e8-8d8e-9909a6e9dce1-kube-api-access-9dtvg\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vrz4g\" (UID: \"a8e50012-20af-48e8-8d8e-9909a6e9dce1\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vrz4g" Nov 03 01:22:20.511962 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:22:20.511527 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnzgs\" (UniqueName: \"kubernetes.io/projected/3ff66d4e-8a7c-4191-8334-03618d2712a2-kube-api-access-jnzgs\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7r5fc\" (UID: \"3ff66d4e-8a7c-4191-8334-03618d2712a2\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7r5fc" Nov 03 01:22:20.511962 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:22:20.511547 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a8e50012-20af-48e8-8d8e-9909a6e9dce1-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vrz4g\" (UID: \"a8e50012-20af-48e8-8d8e-9909a6e9dce1\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vrz4g" Nov 03 01:22:20.511962 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:22:20.511582 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3ff66d4e-8a7c-4191-8334-03618d2712a2-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7r5fc\" (UID: \"3ff66d4e-8a7c-4191-8334-03618d2712a2\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7r5fc" Nov 03 01:22:20.511962 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:22:20.511626 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3ff66d4e-8a7c-4191-8334-03618d2712a2-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7r5fc\" (UID: \"3ff66d4e-8a7c-4191-8334-03618d2712a2\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7r5fc" Nov 03 01:22:20.511962 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:22:20.511657 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a8e50012-20af-48e8-8d8e-9909a6e9dce1-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vrz4g\" (UID: \"a8e50012-20af-48e8-8d8e-9909a6e9dce1\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vrz4g" Nov 03 01:22:20.512220 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:22:20.512002 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a8e50012-20af-48e8-8d8e-9909a6e9dce1-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vrz4g\" (UID: \"a8e50012-20af-48e8-8d8e-9909a6e9dce1\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vrz4g" Nov 03 01:22:20.512220 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:22:20.512017 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a8e50012-20af-48e8-8d8e-9909a6e9dce1-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vrz4g\" (UID: \"a8e50012-20af-48e8-8d8e-9909a6e9dce1\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vrz4g" Nov 03 01:22:20.512220 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:22:20.512036 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3ff66d4e-8a7c-4191-8334-03618d2712a2-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7r5fc\" (UID: \"3ff66d4e-8a7c-4191-8334-03618d2712a2\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7r5fc" Nov 03 01:22:20.512220 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:22:20.512087 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3ff66d4e-8a7c-4191-8334-03618d2712a2-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7r5fc\" (UID: \"3ff66d4e-8a7c-4191-8334-03618d2712a2\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7r5fc" Nov 03 01:22:20.580807 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:22:20.580735 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnzgs\" (UniqueName: \"kubernetes.io/projected/3ff66d4e-8a7c-4191-8334-03618d2712a2-kube-api-access-jnzgs\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7r5fc\" (UID: \"3ff66d4e-8a7c-4191-8334-03618d2712a2\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7r5fc" Nov 03 01:22:20.580807 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:22:20.580733 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dtvg\" (UniqueName: \"kubernetes.io/projected/a8e50012-20af-48e8-8d8e-9909a6e9dce1-kube-api-access-9dtvg\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vrz4g\" (UID: \"a8e50012-20af-48e8-8d8e-9909a6e9dce1\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vrz4g" Nov 03 01:22:20.624444 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:22:20.624409 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7r5fc" Nov 03 01:22:20.631460 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:22:20.631440 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vrz4g" Nov 03 01:22:20.823826 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:22:20.823790 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-85chp"] Nov 03 01:22:20.825936 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:22:20.825914 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-85chp" Nov 03 01:22:20.883185 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:22:20.883155 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 03 01:22:20.883487 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:22:20.883354 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 03 01:22:20.883487 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:22:20.883410 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 03 01:22:20.883487 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:22:20.883453 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 03 01:22:20.883487 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:22:20.883462 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 03 01:22:20.883713 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:22:20.883500 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 03 01:22:20.883832 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:22:20.883804 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-h4j6w\"" Nov 03 01:22:20.887611 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:22:20.887588 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-85chp"] Nov 03 01:22:20.915357 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:22:20.915333 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c1b1cb38-6cd4-41bc-8c11-97951ddbe4af-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-85chp\" (UID: \"c1b1cb38-6cd4-41bc-8c11-97951ddbe4af\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-85chp" Nov 03 01:22:20.915544 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:22:20.915385 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1b1cb38-6cd4-41bc-8c11-97951ddbe4af-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-85chp\" (UID: \"c1b1cb38-6cd4-41bc-8c11-97951ddbe4af\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-85chp" Nov 03 01:22:20.915544 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:22:20.915470 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c1b1cb38-6cd4-41bc-8c11-97951ddbe4af-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-85chp\" (UID: \"c1b1cb38-6cd4-41bc-8c11-97951ddbe4af\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-85chp" Nov 03 01:22:20.915544 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:22:20.915495 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c1b1cb38-6cd4-41bc-8c11-97951ddbe4af-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-85chp\" (UID: \"c1b1cb38-6cd4-41bc-8c11-97951ddbe4af\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-85chp" Nov 03 01:22:20.915544 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:22:20.915527 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c1b1cb38-6cd4-41bc-8c11-97951ddbe4af-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-85chp\" (UID: \"c1b1cb38-6cd4-41bc-8c11-97951ddbe4af\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-85chp" Nov 03 01:22:20.915755 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:22:20.915553 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l299p\" (UniqueName: \"kubernetes.io/projected/c1b1cb38-6cd4-41bc-8c11-97951ddbe4af-kube-api-access-l299p\") pod \"istiod-openshift-gateway-7876bfcb74-85chp\" (UID: \"c1b1cb38-6cd4-41bc-8c11-97951ddbe4af\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-85chp" Nov 03 01:22:20.915755 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:22:20.915676 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1b1cb38-6cd4-41bc-8c11-97951ddbe4af-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-85chp\" (UID: \"c1b1cb38-6cd4-41bc-8c11-97951ddbe4af\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-85chp" Nov 03 01:22:20.991305 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:22:20.991270 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ff66d4e_8a7c_4191_8334_03618d2712a2.slice/crio-1bb3d805bdba416f579d5e9ebc899a8463aa757d7becce6e5e1ee6f3fc40ee39 WatchSource:0}: Error finding container 1bb3d805bdba416f579d5e9ebc899a8463aa757d7becce6e5e1ee6f3fc40ee39: Status 404 returned error can't find the container with id 1bb3d805bdba416f579d5e9ebc899a8463aa757d7becce6e5e1ee6f3fc40ee39 Nov 03 01:22:20.992467 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:22:20.992445 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7r5fc"] Nov 03 01:22:20.993455 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:22:20.993438 5437 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 01:22:21.016402 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:22:21.016374 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1b1cb38-6cd4-41bc-8c11-97951ddbe4af-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-85chp\" (UID: \"c1b1cb38-6cd4-41bc-8c11-97951ddbe4af\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-85chp" Nov 03 01:22:21.016512 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:22:21.016418 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c1b1cb38-6cd4-41bc-8c11-97951ddbe4af-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-85chp\" (UID: \"c1b1cb38-6cd4-41bc-8c11-97951ddbe4af\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-85chp" Nov 03 01:22:21.016609 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:22:21.016589 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c1b1cb38-6cd4-41bc-8c11-97951ddbe4af-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-85chp\" (UID: \"c1b1cb38-6cd4-41bc-8c11-97951ddbe4af\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-85chp" Nov 03 01:22:21.016669 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:22:21.016642 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c1b1cb38-6cd4-41bc-8c11-97951ddbe4af-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-85chp\" (UID: \"c1b1cb38-6cd4-41bc-8c11-97951ddbe4af\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-85chp" Nov 03 01:22:21.016722 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:22:21.016673 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l299p\" (UniqueName: \"kubernetes.io/projected/c1b1cb38-6cd4-41bc-8c11-97951ddbe4af-kube-api-access-l299p\") pod \"istiod-openshift-gateway-7876bfcb74-85chp\" (UID: \"c1b1cb38-6cd4-41bc-8c11-97951ddbe4af\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-85chp" Nov 03 01:22:21.016722 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:22:21.016694 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1b1cb38-6cd4-41bc-8c11-97951ddbe4af-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-85chp\" (UID: \"c1b1cb38-6cd4-41bc-8c11-97951ddbe4af\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-85chp" Nov 03 01:22:21.016829 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:22:21.016744 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c1b1cb38-6cd4-41bc-8c11-97951ddbe4af-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-85chp\" (UID: \"c1b1cb38-6cd4-41bc-8c11-97951ddbe4af\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-85chp" Nov 03 01:22:21.017421 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:22:21.017397 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c1b1cb38-6cd4-41bc-8c11-97951ddbe4af-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-85chp\" (UID: \"c1b1cb38-6cd4-41bc-8c11-97951ddbe4af\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-85chp" Nov 03 01:22:21.018754 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:22:21.018727 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c1b1cb38-6cd4-41bc-8c11-97951ddbe4af-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-85chp\" (UID: \"c1b1cb38-6cd4-41bc-8c11-97951ddbe4af\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-85chp" Nov 03 01:22:21.018835 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:22:21.018809 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c1b1cb38-6cd4-41bc-8c11-97951ddbe4af-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-85chp\" (UID: \"c1b1cb38-6cd4-41bc-8c11-97951ddbe4af\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-85chp" Nov 03 01:22:21.018835 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:22:21.018815 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1b1cb38-6cd4-41bc-8c11-97951ddbe4af-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-85chp\" (UID: \"c1b1cb38-6cd4-41bc-8c11-97951ddbe4af\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-85chp" Nov 03 01:22:21.018912 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:22:21.018895 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c1b1cb38-6cd4-41bc-8c11-97951ddbe4af-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-85chp\" (UID: \"c1b1cb38-6cd4-41bc-8c11-97951ddbe4af\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-85chp" Nov 03 01:22:21.038749 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:22:21.038728 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vrz4g"] Nov 03 01:22:21.039558 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:22:21.039523 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8e50012_20af_48e8_8d8e_9909a6e9dce1.slice/crio-5f7059de60f9308533ea3e4e13f26945752c255dd0e6b138fc6f9bea32a8506e WatchSource:0}: Error finding container 5f7059de60f9308533ea3e4e13f26945752c255dd0e6b138fc6f9bea32a8506e: Status 404 returned error can't find the container with id 5f7059de60f9308533ea3e4e13f26945752c255dd0e6b138fc6f9bea32a8506e Nov 03 01:22:21.079391 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:22:21.079347 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1b1cb38-6cd4-41bc-8c11-97951ddbe4af-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-85chp\" (UID: \"c1b1cb38-6cd4-41bc-8c11-97951ddbe4af\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-85chp" Nov 03 01:22:21.079514 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:22:21.079347 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l299p\" (UniqueName: \"kubernetes.io/projected/c1b1cb38-6cd4-41bc-8c11-97951ddbe4af-kube-api-access-l299p\") pod \"istiod-openshift-gateway-7876bfcb74-85chp\" (UID: \"c1b1cb38-6cd4-41bc-8c11-97951ddbe4af\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-85chp" Nov 03 01:22:21.136639 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:22:21.136565 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-85chp" Nov 03 01:22:21.503120 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:22:21.503088 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1b1cb38_6cd4_41bc_8c11_97951ddbe4af.slice/crio-1b5779f226d83f7e6f651d86bc1cf65cb7c522f713d907a4882426f33db46207 WatchSource:0}: Error finding container 1b5779f226d83f7e6f651d86bc1cf65cb7c522f713d907a4882426f33db46207: Status 404 returned error can't find the container with id 1b5779f226d83f7e6f651d86bc1cf65cb7c522f713d907a4882426f33db46207 Nov 03 01:22:21.504518 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:22:21.504493 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-85chp"] Nov 03 01:22:21.881798 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:22:21.881761 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vrz4g" event={"ID":"a8e50012-20af-48e8-8d8e-9909a6e9dce1","Type":"ContainerStarted","Data":"5f7059de60f9308533ea3e4e13f26945752c255dd0e6b138fc6f9bea32a8506e"} Nov 03 01:22:21.883579 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:22:21.883546 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-85chp" event={"ID":"c1b1cb38-6cd4-41bc-8c11-97951ddbe4af","Type":"ContainerStarted","Data":"1b5779f226d83f7e6f651d86bc1cf65cb7c522f713d907a4882426f33db46207"} Nov 03 01:22:21.887051 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:22:21.885683 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7r5fc" event={"ID":"3ff66d4e-8a7c-4191-8334-03618d2712a2","Type":"ContainerStarted","Data":"1bb3d805bdba416f579d5e9ebc899a8463aa757d7becce6e5e1ee6f3fc40ee39"} Nov 03 01:22:21.998788 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:22:21.998752 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:22:22.210509 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:22:22.210419 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:22:27.591120 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:22:27.591076 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 01:22:27.591544 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:22:27.591169 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 01:22:27.912970 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:22:27.912886 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-85chp" event={"ID":"c1b1cb38-6cd4-41bc-8c11-97951ddbe4af","Type":"ContainerStarted","Data":"a99adae7582e8f0bd8fdd54ef7c8e56fe99ef3188b340cd06e169cd24247ce9a"} Nov 03 01:22:27.913798 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:22:27.913773 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-85chp" Nov 03 01:22:28.090833 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:22:28.090772 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-85chp" podStartSLOduration=2.005915804 podStartE2EDuration="8.090755433s" totalImagesPullingTime="6.084839629s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:22:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:22:28.09059458 +0000 UTC m=+230.120076876" watchObservedRunningTime="2026-11-03 01:22:28.090755433 +0000 UTC m=+230.120237729" Nov 03 01:22:28.918169 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:22:28.918140 5437 generic.go:356] "Generic (PLEG): container finished" podID="a8e50012-20af-48e8-8d8e-9909a6e9dce1" containerID="e9b96e0bff736436d724f758707a9a5b0f35b05ef4eee92c3e5aee3cea36d28d" exitCode=0 Nov 03 01:22:28.918593 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:22:28.918229 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vrz4g" event={"ID":"a8e50012-20af-48e8-8d8e-9909a6e9dce1","Type":"ContainerDied","Data":"e9b96e0bff736436d724f758707a9a5b0f35b05ef4eee92c3e5aee3cea36d28d"} Nov 03 01:22:28.920035 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:22:28.919929 5437 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-85chp container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 01:22:28.920035 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:22:28.920001 5437 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-85chp" podUID="c1b1cb38-6cd4-41bc-8c11-97951ddbe4af" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 01:22:29.158531 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:22:29.158454 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:22:29.414983 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:22:29.414903 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:22:29.795209 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:22:29.795120 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:22:29.922988 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:22:29.922950 5437 generic.go:356] "Generic (PLEG): container finished" podID="3ff66d4e-8a7c-4191-8334-03618d2712a2" containerID="e6f8c1629fdf89fe753a32995cb743c2f617ef457b56d7b79863cca5e4b9f30a" exitCode=0 Nov 03 01:22:29.923505 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:22:29.923041 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7r5fc" event={"ID":"3ff66d4e-8a7c-4191-8334-03618d2712a2","Type":"ContainerDied","Data":"e6f8c1629fdf89fe753a32995cb743c2f617ef457b56d7b79863cca5e4b9f30a"} Nov 03 01:22:29.924240 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:22:29.924092 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-85chp" Nov 03 01:22:30.000893 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:22:30.000851 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:22:30.210996 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:22:30.210964 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:22:33.851279 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:22:33.851153 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:22:34.944959 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:22:34.944850 5437 generic.go:356] "Generic (PLEG): container finished" podID="a8e50012-20af-48e8-8d8e-9909a6e9dce1" containerID="29ded372a069473140354c7d5934237b6748fa11dcbaac064c41aa4da1898f9b" exitCode=0 Nov 03 01:22:34.944959 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:22:34.944945 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vrz4g" event={"ID":"a8e50012-20af-48e8-8d8e-9909a6e9dce1","Type":"ContainerDied","Data":"29ded372a069473140354c7d5934237b6748fa11dcbaac064c41aa4da1898f9b"} Nov 03 01:22:35.950998 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:22:35.950964 5437 generic.go:356] "Generic (PLEG): container finished" podID="3ff66d4e-8a7c-4191-8334-03618d2712a2" containerID="51ca94eae07e1524edf081c150c5514787e33b7787e9e77ea6d68d70fae0188c" exitCode=0 Nov 03 01:22:35.951471 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:22:35.951047 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7r5fc" event={"ID":"3ff66d4e-8a7c-4191-8334-03618d2712a2","Type":"ContainerDied","Data":"51ca94eae07e1524edf081c150c5514787e33b7787e9e77ea6d68d70fae0188c"} Nov 03 01:22:36.894741 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:22:36.894699 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:22:44.038380 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:22:44.038322 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 01:22:44.992813 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:22:44.992779 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vrz4g" event={"ID":"a8e50012-20af-48e8-8d8e-9909a6e9dce1","Type":"ContainerStarted","Data":"3729ea11ff38d31dd86d2414787ec499ca84b8e2663f05fd42422de3574b57b7"} Nov 03 01:22:45.119589 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:22:45.119536 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vrz4g" podStartSLOduration=2.635090935 podStartE2EDuration="25.119523072s" totalImagesPullingTime="22.484432137s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:22:20 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:22:45.118909876 +0000 UTC m=+247.148392173" watchObservedRunningTime="2026-11-03 01:22:45.119523072 +0000 UTC m=+247.149005367" Nov 03 01:22:45.998638 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:22:45.998602 5437 generic.go:356] "Generic (PLEG): container finished" podID="3ff66d4e-8a7c-4191-8334-03618d2712a2" containerID="30656926dc663bdd839db07e559323afeab44d6aa6dc762cacd7ff1b193873bb" exitCode=0 Nov 03 01:22:45.998871 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:22:45.998693 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7r5fc" event={"ID":"3ff66d4e-8a7c-4191-8334-03618d2712a2","Type":"ContainerDied","Data":"30656926dc663bdd839db07e559323afeab44d6aa6dc762cacd7ff1b193873bb"} Nov 03 01:22:46.000744 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:22:46.000711 5437 generic.go:356] "Generic (PLEG): container finished" podID="a8e50012-20af-48e8-8d8e-9909a6e9dce1" containerID="3729ea11ff38d31dd86d2414787ec499ca84b8e2663f05fd42422de3574b57b7" exitCode=0 Nov 03 01:22:46.000849 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:22:46.000744 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vrz4g" event={"ID":"a8e50012-20af-48e8-8d8e-9909a6e9dce1","Type":"ContainerDied","Data":"3729ea11ff38d31dd86d2414787ec499ca84b8e2663f05fd42422de3574b57b7"} Nov 03 01:22:47.133067 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:22:47.133034 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vrz4g" Nov 03 01:22:47.169431 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:22:47.169399 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7r5fc" Nov 03 01:22:47.250169 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:22:47.250137 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8e50012-20af-48e8-8d8e-9909a6e9dce1-util\" (UniqueName: \"kubernetes.io/empty-dir/a8e50012-20af-48e8-8d8e-9909a6e9dce1-util\") pod \"a8e50012-20af-48e8-8d8e-9909a6e9dce1\" (UID: \"a8e50012-20af-48e8-8d8e-9909a6e9dce1\") " Nov 03 01:22:47.250390 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:22:47.250196 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8e50012-20af-48e8-8d8e-9909a6e9dce1-kube-api-access-9dtvg\" (UniqueName: \"kubernetes.io/projected/a8e50012-20af-48e8-8d8e-9909a6e9dce1-kube-api-access-9dtvg\") pod \"a8e50012-20af-48e8-8d8e-9909a6e9dce1\" (UID: \"a8e50012-20af-48e8-8d8e-9909a6e9dce1\") " Nov 03 01:22:47.250390 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:22:47.250271 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ff66d4e-8a7c-4191-8334-03618d2712a2-kube-api-access-jnzgs\" (UniqueName: \"kubernetes.io/projected/3ff66d4e-8a7c-4191-8334-03618d2712a2-kube-api-access-jnzgs\") pod \"3ff66d4e-8a7c-4191-8334-03618d2712a2\" (UID: \"3ff66d4e-8a7c-4191-8334-03618d2712a2\") " Nov 03 01:22:47.250390 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:22:47.250306 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ff66d4e-8a7c-4191-8334-03618d2712a2-bundle\" (UniqueName: \"kubernetes.io/empty-dir/3ff66d4e-8a7c-4191-8334-03618d2712a2-bundle\") pod \"3ff66d4e-8a7c-4191-8334-03618d2712a2\" (UID: \"3ff66d4e-8a7c-4191-8334-03618d2712a2\") " Nov 03 01:22:47.250390 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:22:47.250340 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8e50012-20af-48e8-8d8e-9909a6e9dce1-bundle\" (UniqueName: \"kubernetes.io/empty-dir/a8e50012-20af-48e8-8d8e-9909a6e9dce1-bundle\") pod \"a8e50012-20af-48e8-8d8e-9909a6e9dce1\" (UID: \"a8e50012-20af-48e8-8d8e-9909a6e9dce1\") " Nov 03 01:22:47.250390 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:22:47.250388 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ff66d4e-8a7c-4191-8334-03618d2712a2-util\" (UniqueName: \"kubernetes.io/empty-dir/3ff66d4e-8a7c-4191-8334-03618d2712a2-util\") pod \"3ff66d4e-8a7c-4191-8334-03618d2712a2\" (UID: \"3ff66d4e-8a7c-4191-8334-03618d2712a2\") " Nov 03 01:22:47.251352 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:22:47.251284 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8e50012-20af-48e8-8d8e-9909a6e9dce1-bundle" pod "a8e50012-20af-48e8-8d8e-9909a6e9dce1" (UID: "a8e50012-20af-48e8-8d8e-9909a6e9dce1"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:22:47.252008 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:22:47.251971 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ff66d4e-8a7c-4191-8334-03618d2712a2-bundle" pod "3ff66d4e-8a7c-4191-8334-03618d2712a2" (UID: "3ff66d4e-8a7c-4191-8334-03618d2712a2"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:22:47.253155 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:22:47.253009 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8e50012-20af-48e8-8d8e-9909a6e9dce1-kube-api-access-9dtvg" pod "a8e50012-20af-48e8-8d8e-9909a6e9dce1" (UID: "a8e50012-20af-48e8-8d8e-9909a6e9dce1"). InnerVolumeSpecName "kube-api-access-9dtvg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:22:47.256041 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:22:47.255978 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ff66d4e-8a7c-4191-8334-03618d2712a2-kube-api-access-jnzgs" pod "3ff66d4e-8a7c-4191-8334-03618d2712a2" (UID: "3ff66d4e-8a7c-4191-8334-03618d2712a2"). InnerVolumeSpecName "kube-api-access-jnzgs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:22:47.258667 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:22:47.258628 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ff66d4e-8a7c-4191-8334-03618d2712a2-util" pod "3ff66d4e-8a7c-4191-8334-03618d2712a2" (UID: "3ff66d4e-8a7c-4191-8334-03618d2712a2"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:22:47.258773 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:22:47.258666 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8e50012-20af-48e8-8d8e-9909a6e9dce1-util" pod "a8e50012-20af-48e8-8d8e-9909a6e9dce1" (UID: "a8e50012-20af-48e8-8d8e-9909a6e9dce1"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:22:47.351652 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:22:47.351614 5437 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3ff66d4e-8a7c-4191-8334-03618d2712a2-bundle\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:22:47.351652 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:22:47.351643 5437 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a8e50012-20af-48e8-8d8e-9909a6e9dce1-bundle\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:22:47.351652 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:22:47.351652 5437 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3ff66d4e-8a7c-4191-8334-03618d2712a2-util\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:22:47.351652 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:22:47.351660 5437 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a8e50012-20af-48e8-8d8e-9909a6e9dce1-util\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:22:47.351918 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:22:47.351669 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dtvg\" (UniqueName: \"kubernetes.io/projected/a8e50012-20af-48e8-8d8e-9909a6e9dce1-kube-api-access-9dtvg\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:22:47.351918 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:22:47.351681 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnzgs\" (UniqueName: \"kubernetes.io/projected/3ff66d4e-8a7c-4191-8334-03618d2712a2-kube-api-access-jnzgs\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:22:48.010112 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:22:48.010079 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vrz4g" Nov 03 01:22:48.010300 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:22:48.010082 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vrz4g" event={"ID":"a8e50012-20af-48e8-8d8e-9909a6e9dce1","Type":"ContainerDied","Data":"5f7059de60f9308533ea3e4e13f26945752c255dd0e6b138fc6f9bea32a8506e"} Nov 03 01:22:48.010300 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:22:48.010209 5437 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5f7059de60f9308533ea3e4e13f26945752c255dd0e6b138fc6f9bea32a8506e" Nov 03 01:22:48.011857 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:22:48.011834 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7r5fc" Nov 03 01:22:48.011957 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:22:48.011834 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7r5fc" event={"ID":"3ff66d4e-8a7c-4191-8334-03618d2712a2","Type":"ContainerDied","Data":"1bb3d805bdba416f579d5e9ebc899a8463aa757d7becce6e5e1ee6f3fc40ee39"} Nov 03 01:22:48.011957 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:22:48.011951 5437 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1bb3d805bdba416f579d5e9ebc899a8463aa757d7becce6e5e1ee6f3fc40ee39" Nov 03 01:23:01.915961 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:01.915918 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:02.176678 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:02.176575 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:02.385224 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:02.385188 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:02.638036 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:02.637991 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:02.825028 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:02.824993 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-797bb7c459-hjs7n"] Nov 03 01:23:02.825271 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:02.825258 5437 cpu_manager.go:414] "containerMap: removing container" podUID="a8e50012-20af-48e8-8d8e-9909a6e9dce1" containerName="util" Nov 03 01:23:02.825318 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:02.825272 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8e50012-20af-48e8-8d8e-9909a6e9dce1" containerName="util" Nov 03 01:23:02.825318 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:02.825288 5437 cpu_manager.go:414] "containerMap: removing container" podUID="a8e50012-20af-48e8-8d8e-9909a6e9dce1" containerName="pull" Nov 03 01:23:02.825318 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:02.825293 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8e50012-20af-48e8-8d8e-9909a6e9dce1" containerName="pull" Nov 03 01:23:02.825318 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:02.825299 5437 cpu_manager.go:414] "containerMap: removing container" podUID="3ff66d4e-8a7c-4191-8334-03618d2712a2" containerName="extract" Nov 03 01:23:02.825318 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:02.825305 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ff66d4e-8a7c-4191-8334-03618d2712a2" containerName="extract" Nov 03 01:23:02.825517 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:02.825325 5437 cpu_manager.go:414] "containerMap: removing container" podUID="3ff66d4e-8a7c-4191-8334-03618d2712a2" containerName="util" Nov 03 01:23:02.825517 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:02.825330 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ff66d4e-8a7c-4191-8334-03618d2712a2" containerName="util" Nov 03 01:23:02.825517 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:02.825335 5437 cpu_manager.go:414] "containerMap: removing container" podUID="3ff66d4e-8a7c-4191-8334-03618d2712a2" containerName="pull" Nov 03 01:23:02.825517 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:02.825340 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ff66d4e-8a7c-4191-8334-03618d2712a2" containerName="pull" Nov 03 01:23:02.825517 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:02.825347 5437 cpu_manager.go:414] "containerMap: removing container" podUID="a8e50012-20af-48e8-8d8e-9909a6e9dce1" containerName="extract" Nov 03 01:23:02.825517 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:02.825351 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8e50012-20af-48e8-8d8e-9909a6e9dce1" containerName="extract" Nov 03 01:23:02.825517 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:02.825431 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8e50012-20af-48e8-8d8e-9909a6e9dce1" containerName="extract" Nov 03 01:23:02.825517 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:02.825440 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ff66d4e-8a7c-4191-8334-03618d2712a2" containerName="extract" Nov 03 01:23:02.827203 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:02.827186 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-797bb7c459-hjs7n" Nov 03 01:23:02.882495 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:02.882454 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 03 01:23:02.917812 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:02.917723 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:02.933637 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:02.933595 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 03 01:23:02.933637 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:02.933615 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 03 01:23:02.933637 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:02.933626 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-rqx2g\"" Nov 03 01:23:02.933637 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:02.933598 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 03 01:23:02.933958 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:02.933616 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 03 01:23:02.935004 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:02.934982 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-797bb7c459-hjs7n"] Nov 03 01:23:02.972049 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:02.972016 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f333e6bc-3bc2-41b3-9cea-a203f0f3efc4-manifests\") pod \"opendatahub-operator-controller-manager-797bb7c459-hjs7n\" (UID: \"f333e6bc-3bc2-41b3-9cea-a203f0f3efc4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-797bb7c459-hjs7n" Nov 03 01:23:02.972049 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:02.972054 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzxgc\" (UniqueName: \"kubernetes.io/projected/f333e6bc-3bc2-41b3-9cea-a203f0f3efc4-kube-api-access-gzxgc\") pod \"opendatahub-operator-controller-manager-797bb7c459-hjs7n\" (UID: \"f333e6bc-3bc2-41b3-9cea-a203f0f3efc4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-797bb7c459-hjs7n" Nov 03 01:23:02.972329 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:02.972080 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f333e6bc-3bc2-41b3-9cea-a203f0f3efc4-tmp\") pod \"opendatahub-operator-controller-manager-797bb7c459-hjs7n\" (UID: \"f333e6bc-3bc2-41b3-9cea-a203f0f3efc4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-797bb7c459-hjs7n" Nov 03 01:23:02.972329 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:02.972193 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f333e6bc-3bc2-41b3-9cea-a203f0f3efc4-webhook-cert\") pod \"opendatahub-operator-controller-manager-797bb7c459-hjs7n\" (UID: \"f333e6bc-3bc2-41b3-9cea-a203f0f3efc4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-797bb7c459-hjs7n" Nov 03 01:23:02.972329 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:02.972245 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f333e6bc-3bc2-41b3-9cea-a203f0f3efc4-metrics-cert\") pod \"opendatahub-operator-controller-manager-797bb7c459-hjs7n\" (UID: \"f333e6bc-3bc2-41b3-9cea-a203f0f3efc4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-797bb7c459-hjs7n" Nov 03 01:23:02.972329 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:02.972295 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f333e6bc-3bc2-41b3-9cea-a203f0f3efc4-apiservice-cert\") pod \"opendatahub-operator-controller-manager-797bb7c459-hjs7n\" (UID: \"f333e6bc-3bc2-41b3-9cea-a203f0f3efc4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-797bb7c459-hjs7n" Nov 03 01:23:03.073738 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:03.073694 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f333e6bc-3bc2-41b3-9cea-a203f0f3efc4-manifests\") pod \"opendatahub-operator-controller-manager-797bb7c459-hjs7n\" (UID: \"f333e6bc-3bc2-41b3-9cea-a203f0f3efc4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-797bb7c459-hjs7n" Nov 03 01:23:03.073738 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:03.073742 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzxgc\" (UniqueName: \"kubernetes.io/projected/f333e6bc-3bc2-41b3-9cea-a203f0f3efc4-kube-api-access-gzxgc\") pod \"opendatahub-operator-controller-manager-797bb7c459-hjs7n\" (UID: \"f333e6bc-3bc2-41b3-9cea-a203f0f3efc4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-797bb7c459-hjs7n" Nov 03 01:23:03.074012 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:03.073768 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f333e6bc-3bc2-41b3-9cea-a203f0f3efc4-tmp\") pod \"opendatahub-operator-controller-manager-797bb7c459-hjs7n\" (UID: \"f333e6bc-3bc2-41b3-9cea-a203f0f3efc4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-797bb7c459-hjs7n" Nov 03 01:23:03.074012 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:03.073794 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f333e6bc-3bc2-41b3-9cea-a203f0f3efc4-webhook-cert\") pod \"opendatahub-operator-controller-manager-797bb7c459-hjs7n\" (UID: \"f333e6bc-3bc2-41b3-9cea-a203f0f3efc4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-797bb7c459-hjs7n" Nov 03 01:23:03.074012 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:03.073812 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f333e6bc-3bc2-41b3-9cea-a203f0f3efc4-metrics-cert\") pod \"opendatahub-operator-controller-manager-797bb7c459-hjs7n\" (UID: \"f333e6bc-3bc2-41b3-9cea-a203f0f3efc4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-797bb7c459-hjs7n" Nov 03 01:23:03.074012 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:03.073826 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f333e6bc-3bc2-41b3-9cea-a203f0f3efc4-apiservice-cert\") pod \"opendatahub-operator-controller-manager-797bb7c459-hjs7n\" (UID: \"f333e6bc-3bc2-41b3-9cea-a203f0f3efc4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-797bb7c459-hjs7n" Nov 03 01:23:03.074220 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:03.074109 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f333e6bc-3bc2-41b3-9cea-a203f0f3efc4-manifests\") pod \"opendatahub-operator-controller-manager-797bb7c459-hjs7n\" (UID: \"f333e6bc-3bc2-41b3-9cea-a203f0f3efc4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-797bb7c459-hjs7n" Nov 03 01:23:03.076133 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:03.076091 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f333e6bc-3bc2-41b3-9cea-a203f0f3efc4-tmp\") pod \"opendatahub-operator-controller-manager-797bb7c459-hjs7n\" (UID: \"f333e6bc-3bc2-41b3-9cea-a203f0f3efc4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-797bb7c459-hjs7n" Nov 03 01:23:03.076324 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:03.076303 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f333e6bc-3bc2-41b3-9cea-a203f0f3efc4-metrics-cert\") pod \"opendatahub-operator-controller-manager-797bb7c459-hjs7n\" (UID: \"f333e6bc-3bc2-41b3-9cea-a203f0f3efc4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-797bb7c459-hjs7n" Nov 03 01:23:03.076433 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:03.076413 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f333e6bc-3bc2-41b3-9cea-a203f0f3efc4-apiservice-cert\") pod \"opendatahub-operator-controller-manager-797bb7c459-hjs7n\" (UID: \"f333e6bc-3bc2-41b3-9cea-a203f0f3efc4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-797bb7c459-hjs7n" Nov 03 01:23:03.076497 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:03.076427 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f333e6bc-3bc2-41b3-9cea-a203f0f3efc4-webhook-cert\") pod \"opendatahub-operator-controller-manager-797bb7c459-hjs7n\" (UID: \"f333e6bc-3bc2-41b3-9cea-a203f0f3efc4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-797bb7c459-hjs7n" Nov 03 01:23:03.134807 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:03.134766 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:03.140391 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:03.140340 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzxgc\" (UniqueName: \"kubernetes.io/projected/f333e6bc-3bc2-41b3-9cea-a203f0f3efc4-kube-api-access-gzxgc\") pod \"opendatahub-operator-controller-manager-797bb7c459-hjs7n\" (UID: \"f333e6bc-3bc2-41b3-9cea-a203f0f3efc4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-797bb7c459-hjs7n" Nov 03 01:23:03.396550 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:03.396505 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:03.440049 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:03.440004 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-797bb7c459-hjs7n" Nov 03 01:23:03.604702 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:03.604655 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:04.063112 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:23:04.063082 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf333e6bc_3bc2_41b3_9cea_a203f0f3efc4.slice/crio-31d44356cd7bfd94e21cfdc9a23e2252538f80c2b88648a445c2afb313e21754 WatchSource:0}: Error finding container 31d44356cd7bfd94e21cfdc9a23e2252538f80c2b88648a445c2afb313e21754: Status 404 returned error can't find the container with id 31d44356cd7bfd94e21cfdc9a23e2252538f80c2b88648a445c2afb313e21754 Nov 03 01:23:04.068422 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:04.068356 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-797bb7c459-hjs7n" event={"ID":"f333e6bc-3bc2-41b3-9cea-a203f0f3efc4","Type":"ContainerStarted","Data":"31d44356cd7bfd94e21cfdc9a23e2252538f80c2b88648a445c2afb313e21754"} Nov 03 01:23:04.069813 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:04.069791 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-797bb7c459-hjs7n"] Nov 03 01:23:05.814171 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:05.814114 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:06.068244 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:06.068096 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:06.369874 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:06.369789 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:06.581429 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:06.581393 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:08.086815 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:08.086720 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-797bb7c459-hjs7n" event={"ID":"f333e6bc-3bc2-41b3-9cea-a203f0f3efc4","Type":"ContainerStarted","Data":"8f62621c70bd8849b316a3dcb8b85f4ac4d18eee948bd0f9f4aff910135c52f6"} Nov 03 01:23:09.092116 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:09.092080 5437 generic.go:356] "Generic (PLEG): container finished" podID="f333e6bc-3bc2-41b3-9cea-a203f0f3efc4" containerID="8f62621c70bd8849b316a3dcb8b85f4ac4d18eee948bd0f9f4aff910135c52f6" exitCode=0 Nov 03 01:23:09.092556 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:09.092151 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-797bb7c459-hjs7n" event={"ID":"f333e6bc-3bc2-41b3-9cea-a203f0f3efc4","Type":"ContainerDied","Data":"8f62621c70bd8849b316a3dcb8b85f4ac4d18eee948bd0f9f4aff910135c52f6"} Nov 03 01:23:10.097633 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:10.097597 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-797bb7c459-hjs7n" event={"ID":"f333e6bc-3bc2-41b3-9cea-a203f0f3efc4","Type":"ContainerStarted","Data":"3f4f1d99cd939709cf7245fa0db91914ad0e06f7cfe57f8e0c341fbb6fb86293"} Nov 03 01:23:10.098032 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:10.097714 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-797bb7c459-hjs7n" Nov 03 01:23:10.296726 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:10.296642 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-797bb7c459-hjs7n" podStartSLOduration=3.204060611 podStartE2EDuration="8.296627156s" totalImagesPullingTime="4.628440639s" totalInitContainerRuntime="464.125906ms" isStatefulPod=false podCreationTimestamp="2026-11-03 01:23:02 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:23:10.247357381 +0000 UTC m=+272.276839667" watchObservedRunningTime="2026-11-03 01:23:10.296627156 +0000 UTC m=+272.326109442" Nov 03 01:23:11.603387 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:11.603327 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:11.810921 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:11.810881 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:14.034708 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:23:14.034665 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 01:23:14.212412 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:14.212373 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:16.944573 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:16.944484 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:18.658632 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:18.658598 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:21.106353 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:21.106318 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-797bb7c459-hjs7n" Nov 03 01:23:21.354206 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:21.354163 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:22.150514 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:22.150481 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:22.977838 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:22.977801 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:24.817015 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:24.816975 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:26.279587 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:26.279543 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:26.531696 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:26.531608 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:27.549899 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:27.549856 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:28.145830 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:28.145792 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:28.687702 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:28.687669 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:29.165834 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:29.165801 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:31.389209 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:31.389175 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:31.757234 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:31.757144 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:32.662645 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:32.662609 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:32.918694 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:32.918601 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:33.288906 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:33.288823 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:33.500565 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:33.500529 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:33.967820 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:33.967785 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-zlql6"] Nov 03 01:23:33.969896 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:33.969875 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-zlql6" Nov 03 01:23:33.990516 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:33.990484 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgwbw\" (UniqueName: \"kubernetes.io/projected/1447e14e-f0a8-43bd-8e05-9d32b57fcf34-kube-api-access-sgwbw\") pod \"dns-operator-controller-manager-8557cbdb84-zlql6\" (UID: \"1447e14e-f0a8-43bd-8e05-9d32b57fcf34\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-zlql6" Nov 03 01:23:34.030011 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:34.029975 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-4c6bw\"" Nov 03 01:23:34.032459 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:34.032439 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Nov 03 01:23:34.078645 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:34.078605 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 03 01:23:34.078645 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:34.078623 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 03 01:23:34.079515 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:34.079492 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-zlql6"] Nov 03 01:23:34.092043 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:34.091989 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgwbw\" (UniqueName: \"kubernetes.io/projected/1447e14e-f0a8-43bd-8e05-9d32b57fcf34-kube-api-access-sgwbw\") pod \"dns-operator-controller-manager-8557cbdb84-zlql6\" (UID: \"1447e14e-f0a8-43bd-8e05-9d32b57fcf34\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-zlql6" Nov 03 01:23:34.154506 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:34.154473 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgwbw\" (UniqueName: \"kubernetes.io/projected/1447e14e-f0a8-43bd-8e05-9d32b57fcf34-kube-api-access-sgwbw\") pod \"dns-operator-controller-manager-8557cbdb84-zlql6\" (UID: \"1447e14e-f0a8-43bd-8e05-9d32b57fcf34\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-zlql6" Nov 03 01:23:34.281759 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:34.281661 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-zlql6" Nov 03 01:23:34.552913 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:34.552872 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:34.577646 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:34.577612 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-68dcf9949f-hcw7m"] Nov 03 01:23:34.580383 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:34.580342 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-68dcf9949f-hcw7m" Nov 03 01:23:34.594983 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:34.594949 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dt7mw\" (UniqueName: \"kubernetes.io/projected/64c2563a-4b6a-4d94-8cb7-09fce6d2e734-kube-api-access-dt7mw\") pod \"kube-auth-proxy-68dcf9949f-hcw7m\" (UID: \"64c2563a-4b6a-4d94-8cb7-09fce6d2e734\") " pod="openshift-ingress/kube-auth-proxy-68dcf9949f-hcw7m" Nov 03 01:23:34.595128 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:34.594998 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/64c2563a-4b6a-4d94-8cb7-09fce6d2e734-tmp\") pod \"kube-auth-proxy-68dcf9949f-hcw7m\" (UID: \"64c2563a-4b6a-4d94-8cb7-09fce6d2e734\") " pod="openshift-ingress/kube-auth-proxy-68dcf9949f-hcw7m" Nov 03 01:23:34.595128 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:34.595044 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64c2563a-4b6a-4d94-8cb7-09fce6d2e734-tls-certs\") pod \"kube-auth-proxy-68dcf9949f-hcw7m\" (UID: \"64c2563a-4b6a-4d94-8cb7-09fce6d2e734\") " pod="openshift-ingress/kube-auth-proxy-68dcf9949f-hcw7m" Nov 03 01:23:34.648621 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:34.648591 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-b6fzt\"" Nov 03 01:23:34.648621 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:34.648591 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Nov 03 01:23:34.648621 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:34.648591 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Nov 03 01:23:34.667410 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:34.667376 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-68dcf9949f-hcw7m"] Nov 03 01:23:34.696326 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:34.696285 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dt7mw\" (UniqueName: \"kubernetes.io/projected/64c2563a-4b6a-4d94-8cb7-09fce6d2e734-kube-api-access-dt7mw\") pod \"kube-auth-proxy-68dcf9949f-hcw7m\" (UID: \"64c2563a-4b6a-4d94-8cb7-09fce6d2e734\") " pod="openshift-ingress/kube-auth-proxy-68dcf9949f-hcw7m" Nov 03 01:23:34.696539 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:34.696347 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/64c2563a-4b6a-4d94-8cb7-09fce6d2e734-tmp\") pod \"kube-auth-proxy-68dcf9949f-hcw7m\" (UID: \"64c2563a-4b6a-4d94-8cb7-09fce6d2e734\") " pod="openshift-ingress/kube-auth-proxy-68dcf9949f-hcw7m" Nov 03 01:23:34.696539 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:34.696444 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64c2563a-4b6a-4d94-8cb7-09fce6d2e734-tls-certs\") pod \"kube-auth-proxy-68dcf9949f-hcw7m\" (UID: \"64c2563a-4b6a-4d94-8cb7-09fce6d2e734\") " pod="openshift-ingress/kube-auth-proxy-68dcf9949f-hcw7m" Nov 03 01:23:34.698871 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:34.698844 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/64c2563a-4b6a-4d94-8cb7-09fce6d2e734-tmp\") pod \"kube-auth-proxy-68dcf9949f-hcw7m\" (UID: \"64c2563a-4b6a-4d94-8cb7-09fce6d2e734\") " pod="openshift-ingress/kube-auth-proxy-68dcf9949f-hcw7m" Nov 03 01:23:34.699013 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:34.698991 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64c2563a-4b6a-4d94-8cb7-09fce6d2e734-tls-certs\") pod \"kube-auth-proxy-68dcf9949f-hcw7m\" (UID: \"64c2563a-4b6a-4d94-8cb7-09fce6d2e734\") " pod="openshift-ingress/kube-auth-proxy-68dcf9949f-hcw7m" Nov 03 01:23:34.701718 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:34.701694 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-zlql6"] Nov 03 01:23:34.702192 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:23:34.702161 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1447e14e_f0a8_43bd_8e05_9d32b57fcf34.slice/crio-d6402031a61d998ef4f70d8d96ef69e5d1ede9ebad3e5e48729eb0e628aaeef6 WatchSource:0}: Error finding container d6402031a61d998ef4f70d8d96ef69e5d1ede9ebad3e5e48729eb0e628aaeef6: Status 404 returned error can't find the container with id d6402031a61d998ef4f70d8d96ef69e5d1ede9ebad3e5e48729eb0e628aaeef6 Nov 03 01:23:34.765604 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:34.765546 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dt7mw\" (UniqueName: \"kubernetes.io/projected/64c2563a-4b6a-4d94-8cb7-09fce6d2e734-kube-api-access-dt7mw\") pod \"kube-auth-proxy-68dcf9949f-hcw7m\" (UID: \"64c2563a-4b6a-4d94-8cb7-09fce6d2e734\") " pod="openshift-ingress/kube-auth-proxy-68dcf9949f-hcw7m" Nov 03 01:23:34.838289 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:34.838200 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:34.892750 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:34.892716 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-68dcf9949f-hcw7m" Nov 03 01:23:35.193611 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:35.193524 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-zlql6" event={"ID":"1447e14e-f0a8-43bd-8e05-9d32b57fcf34","Type":"ContainerStarted","Data":"d6402031a61d998ef4f70d8d96ef69e5d1ede9ebad3e5e48729eb0e628aaeef6"} Nov 03 01:23:35.266043 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:23:35.266009 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64c2563a_4b6a_4d94_8cb7_09fce6d2e734.slice/crio-bcd1ebc2ca5f234828122771f848becff6945dd65621ddccf7e2feff6a22da58 WatchSource:0}: Error finding container bcd1ebc2ca5f234828122771f848becff6945dd65621ddccf7e2feff6a22da58: Status 404 returned error can't find the container with id bcd1ebc2ca5f234828122771f848becff6945dd65621ddccf7e2feff6a22da58 Nov 03 01:23:35.268580 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:35.268553 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-68dcf9949f-hcw7m"] Nov 03 01:23:35.514720 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:35.514642 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:35.737537 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:35.737499 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:35.975818 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:35.975780 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:36.190963 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:36.190930 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:36.198728 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:36.198695 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-68dcf9949f-hcw7m" event={"ID":"64c2563a-4b6a-4d94-8cb7-09fce6d2e734","Type":"ContainerStarted","Data":"bcd1ebc2ca5f234828122771f848becff6945dd65621ddccf7e2feff6a22da58"} Nov 03 01:23:37.253638 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:37.253557 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:38.614899 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:38.614851 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:38.633051 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:38.632716 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-8hpgc"] Nov 03 01:23:38.638721 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:38.638691 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-8hpgc" Nov 03 01:23:38.689651 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:38.689562 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-8hpgc"] Nov 03 01:23:38.733013 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:38.732920 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7njh\" (UniqueName: \"kubernetes.io/projected/8c510af4-a6fe-4848-a1b6-704aea97423c-kube-api-access-d7njh\") pod \"limitador-limitador-74dfcfb894-8hpgc\" (UID: \"8c510af4-a6fe-4848-a1b6-704aea97423c\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-8hpgc" Nov 03 01:23:38.733215 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:38.733074 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8c510af4-a6fe-4848-a1b6-704aea97423c-config-file\") pod \"limitador-limitador-74dfcfb894-8hpgc\" (UID: \"8c510af4-a6fe-4848-a1b6-704aea97423c\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-8hpgc" Nov 03 01:23:38.740680 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:38.740646 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-8vbrr\"" Nov 03 01:23:38.740680 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:38.740655 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Nov 03 01:23:38.834080 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:38.833690 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8c510af4-a6fe-4848-a1b6-704aea97423c-config-file\") pod \"limitador-limitador-74dfcfb894-8hpgc\" (UID: \"8c510af4-a6fe-4848-a1b6-704aea97423c\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-8hpgc" Nov 03 01:23:38.834080 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:38.833764 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7njh\" (UniqueName: \"kubernetes.io/projected/8c510af4-a6fe-4848-a1b6-704aea97423c-kube-api-access-d7njh\") pod \"limitador-limitador-74dfcfb894-8hpgc\" (UID: \"8c510af4-a6fe-4848-a1b6-704aea97423c\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-8hpgc" Nov 03 01:23:38.834589 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:38.834546 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8c510af4-a6fe-4848-a1b6-704aea97423c-config-file\") pod \"limitador-limitador-74dfcfb894-8hpgc\" (UID: \"8c510af4-a6fe-4848-a1b6-704aea97423c\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-8hpgc" Nov 03 01:23:38.885994 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:38.885912 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-8hpgc"] Nov 03 01:23:38.891637 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:38.891591 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:38.898500 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:38.898472 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7njh\" (UniqueName: \"kubernetes.io/projected/8c510af4-a6fe-4848-a1b6-704aea97423c-kube-api-access-d7njh\") pod \"limitador-limitador-74dfcfb894-8hpgc\" (UID: \"8c510af4-a6fe-4848-a1b6-704aea97423c\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-8hpgc" Nov 03 01:23:38.979797 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:38.979756 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-8hpgc" Nov 03 01:23:39.089823 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:39.089791 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:39.132541 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:39.132515 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Nov 03 01:23:39.172904 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:39.172863 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Nov 03 01:23:39.215862 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:39.215833 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-zlql6" event={"ID":"1447e14e-f0a8-43bd-8e05-9d32b57fcf34","Type":"ContainerStarted","Data":"63479b49244c4d1da01a6556d603d2d2241c1a80139665eb0ab46c9fcc3c2e6c"} Nov 03 01:23:39.216011 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:39.215943 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-zlql6" Nov 03 01:23:39.335696 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:39.335635 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-zlql6" podStartSLOduration=1.961329334 podStartE2EDuration="6.335617866s" totalImagesPullingTime="4.374288532s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:23:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:23:39.335134431 +0000 UTC m=+301.364616751" watchObservedRunningTime="2026-11-03 01:23:39.335617866 +0000 UTC m=+301.365100161" Nov 03 01:23:39.354972 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:23:39.354932 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c510af4_a6fe_4848_a1b6_704aea97423c.slice/crio-19a42041ddbf93617bbbe2c9222961eda20148e49aeaf78e7b2272944f8e2e21 WatchSource:0}: Error finding container 19a42041ddbf93617bbbe2c9222961eda20148e49aeaf78e7b2272944f8e2e21: Status 404 returned error can't find the container with id 19a42041ddbf93617bbbe2c9222961eda20148e49aeaf78e7b2272944f8e2e21 Nov 03 01:23:39.356837 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:39.356817 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-8hpgc"] Nov 03 01:23:39.868347 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:39.868303 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:40.071763 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:40.071723 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:40.221295 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:40.221196 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-8hpgc" event={"ID":"8c510af4-a6fe-4848-a1b6-704aea97423c","Type":"ContainerStarted","Data":"19a42041ddbf93617bbbe2c9222961eda20148e49aeaf78e7b2272944f8e2e21"} Nov 03 01:23:40.222607 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:40.222578 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-68dcf9949f-hcw7m" event={"ID":"64c2563a-4b6a-4d94-8cb7-09fce6d2e734","Type":"ContainerStarted","Data":"e76777278033bafc95b9814dd63292acf074a40c2a407be17e6fc484b9610dd6"} Nov 03 01:23:40.354342 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:40.354150 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:40.360853 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:40.360798 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-68dcf9949f-hcw7m" podStartSLOduration=2.510461371 podStartE2EDuration="6.360782474s" totalImagesPullingTime="3.850321103s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:23:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:23:40.347790766 +0000 UTC m=+302.377273072" watchObservedRunningTime="2026-11-03 01:23:40.360782474 +0000 UTC m=+302.390264769" Nov 03 01:23:40.559838 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:40.559805 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:41.760080 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:41.760047 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:42.135835 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:42.135803 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:42.232034 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:42.231995 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-8hpgc" event={"ID":"8c510af4-a6fe-4848-a1b6-704aea97423c","Type":"ContainerStarted","Data":"5b38fc134e0971ebd09a906f234bb77a0fc5c470aaf1ccb96d950c515d3f9185"} Nov 03 01:23:42.232232 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:42.232109 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-8hpgc" Nov 03 01:23:42.351597 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:42.351538 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-8hpgc" podStartSLOduration=2.108216942 podStartE2EDuration="4.351525297s" totalImagesPullingTime="2.243308355s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:23:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:23:42.351071239 +0000 UTC m=+304.380553552" watchObservedRunningTime="2026-11-03 01:23:42.351525297 +0000 UTC m=+304.381007593" Nov 03 01:23:42.675758 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:42.675727 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:44.034474 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:23:44.034440 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 01:23:48.157815 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:48.157773 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:48.381462 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:48.381416 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:50.225435 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:50.225404 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-zlql6" Nov 03 01:23:50.500614 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:50.500527 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:50.762499 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:50.762410 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:53.237508 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:53.237479 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-8hpgc" Nov 03 01:23:54.849544 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:54.849491 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:55.055207 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:55.055159 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:56.247376 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:56.247338 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-8hpgc"] Nov 03 01:23:56.247813 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:56.247547 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-8hpgc" podUID="8c510af4-a6fe-4848-a1b6-704aea97423c" containerName="limitador" containerID="cri-o://5b38fc134e0971ebd09a906f234bb77a0fc5c470aaf1ccb96d950c515d3f9185" gracePeriod=30 Nov 03 01:23:56.794334 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:56.794309 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-8hpgc" Nov 03 01:23:56.896024 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:56.895929 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c510af4-a6fe-4848-a1b6-704aea97423c-kube-api-access-d7njh\" (UniqueName: \"kubernetes.io/projected/8c510af4-a6fe-4848-a1b6-704aea97423c-kube-api-access-d7njh\") pod \"8c510af4-a6fe-4848-a1b6-704aea97423c\" (UID: \"8c510af4-a6fe-4848-a1b6-704aea97423c\") " Nov 03 01:23:56.896024 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:56.896016 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c510af4-a6fe-4848-a1b6-704aea97423c-config-file\" (UniqueName: \"kubernetes.io/configmap/8c510af4-a6fe-4848-a1b6-704aea97423c-config-file\") pod \"8c510af4-a6fe-4848-a1b6-704aea97423c\" (UID: \"8c510af4-a6fe-4848-a1b6-704aea97423c\") " Nov 03 01:23:56.896448 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:56.896416 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c510af4-a6fe-4848-a1b6-704aea97423c-config-file" pod "8c510af4-a6fe-4848-a1b6-704aea97423c" (UID: "8c510af4-a6fe-4848-a1b6-704aea97423c"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:23:56.898203 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:56.898181 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c510af4-a6fe-4848-a1b6-704aea97423c-kube-api-access-d7njh" pod "8c510af4-a6fe-4848-a1b6-704aea97423c" (UID: "8c510af4-a6fe-4848-a1b6-704aea97423c"). InnerVolumeSpecName "kube-api-access-d7njh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:23:56.997286 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:56.997227 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7njh\" (UniqueName: \"kubernetes.io/projected/8c510af4-a6fe-4848-a1b6-704aea97423c-kube-api-access-d7njh\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:23:56.997286 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:56.997275 5437 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8c510af4-a6fe-4848-a1b6-704aea97423c-config-file\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:23:57.293695 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:57.293608 5437 generic.go:356] "Generic (PLEG): container finished" podID="8c510af4-a6fe-4848-a1b6-704aea97423c" containerID="5b38fc134e0971ebd09a906f234bb77a0fc5c470aaf1ccb96d950c515d3f9185" exitCode=0 Nov 03 01:23:57.293695 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:57.293654 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-8hpgc" Nov 03 01:23:57.294172 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:57.293695 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-8hpgc" event={"ID":"8c510af4-a6fe-4848-a1b6-704aea97423c","Type":"ContainerDied","Data":"5b38fc134e0971ebd09a906f234bb77a0fc5c470aaf1ccb96d950c515d3f9185"} Nov 03 01:23:57.294172 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:57.293748 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-8hpgc" event={"ID":"8c510af4-a6fe-4848-a1b6-704aea97423c","Type":"ContainerDied","Data":"19a42041ddbf93617bbbe2c9222961eda20148e49aeaf78e7b2272944f8e2e21"} Nov 03 01:23:57.294172 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:57.293772 5437 scope.go:122] "RemoveContainer" containerID="5b38fc134e0971ebd09a906f234bb77a0fc5c470aaf1ccb96d950c515d3f9185" Nov 03 01:23:57.302294 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:57.302271 5437 scope.go:122] "RemoveContainer" containerID="5b38fc134e0971ebd09a906f234bb77a0fc5c470aaf1ccb96d950c515d3f9185" Nov 03 01:23:57.302634 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:23:57.302613 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b38fc134e0971ebd09a906f234bb77a0fc5c470aaf1ccb96d950c515d3f9185\": container with ID starting with 5b38fc134e0971ebd09a906f234bb77a0fc5c470aaf1ccb96d950c515d3f9185 not found: ID does not exist" containerID="5b38fc134e0971ebd09a906f234bb77a0fc5c470aaf1ccb96d950c515d3f9185" Nov 03 01:23:57.302700 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:57.302646 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b38fc134e0971ebd09a906f234bb77a0fc5c470aaf1ccb96d950c515d3f9185"} err="failed to get container status \"5b38fc134e0971ebd09a906f234bb77a0fc5c470aaf1ccb96d950c515d3f9185\": rpc error: code = NotFound desc = could not find container \"5b38fc134e0971ebd09a906f234bb77a0fc5c470aaf1ccb96d950c515d3f9185\": container with ID starting with 5b38fc134e0971ebd09a906f234bb77a0fc5c470aaf1ccb96d950c515d3f9185 not found: ID does not exist" Nov 03 01:23:57.464830 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:57.464791 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-8hpgc"] Nov 03 01:23:57.516500 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:57.516466 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-8hpgc"] Nov 03 01:23:58.087348 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:58.087313 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c510af4-a6fe-4848-a1b6-704aea97423c" path="/var/lib/kubelet/pods/8c510af4-a6fe-4848-a1b6-704aea97423c/volumes" Nov 03 01:23:58.455200 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:58.455124 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:58.704285 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:58.704252 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:58.911273 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:58.911232 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:59.114389 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:23:59.114335 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:24:00.375372 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:24:00.375331 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:24:00.626711 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:24:00.626630 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:24:01.092139 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:24:01.092105 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-529bb"] Nov 03 01:24:01.092435 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:24:01.092421 5437 cpu_manager.go:414] "containerMap: removing container" podUID="8c510af4-a6fe-4848-a1b6-704aea97423c" containerName="limitador" Nov 03 01:24:01.092435 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:24:01.092435 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c510af4-a6fe-4848-a1b6-704aea97423c" containerName="limitador" Nov 03 01:24:01.092557 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:24:01.092495 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c510af4-a6fe-4848-a1b6-704aea97423c" containerName="limitador" Nov 03 01:24:01.094979 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:24:01.094960 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-529bb" Nov 03 01:24:01.149745 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:24:01.149714 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-529bb"] Nov 03 01:24:01.200582 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:24:01.200536 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 01:24:01.200582 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:24:01.200559 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 01:24:01.200807 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:24:01.200607 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Nov 03 01:24:01.200807 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:24:01.200637 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-wgbsz\"" Nov 03 01:24:01.238706 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:24:01.238675 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:24:01.239673 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:24:01.239650 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96kxx\" (UniqueName: \"kubernetes.io/projected/096e923b-f4be-4e2b-913f-a2ecb011a0c0-kube-api-access-96kxx\") pod \"postgres-7cdbf6dc44-529bb\" (UID: \"096e923b-f4be-4e2b-913f-a2ecb011a0c0\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-529bb" Nov 03 01:24:01.239737 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:24:01.239682 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/096e923b-f4be-4e2b-913f-a2ecb011a0c0-data\") pod \"postgres-7cdbf6dc44-529bb\" (UID: \"096e923b-f4be-4e2b-913f-a2ecb011a0c0\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-529bb" Nov 03 01:24:01.340378 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:24:01.340335 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96kxx\" (UniqueName: \"kubernetes.io/projected/096e923b-f4be-4e2b-913f-a2ecb011a0c0-kube-api-access-96kxx\") pod \"postgres-7cdbf6dc44-529bb\" (UID: \"096e923b-f4be-4e2b-913f-a2ecb011a0c0\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-529bb" Nov 03 01:24:01.340566 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:24:01.340406 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/096e923b-f4be-4e2b-913f-a2ecb011a0c0-data\") pod \"postgres-7cdbf6dc44-529bb\" (UID: \"096e923b-f4be-4e2b-913f-a2ecb011a0c0\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-529bb" Nov 03 01:24:01.340847 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:24:01.340825 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/096e923b-f4be-4e2b-913f-a2ecb011a0c0-data\") pod \"postgres-7cdbf6dc44-529bb\" (UID: \"096e923b-f4be-4e2b-913f-a2ecb011a0c0\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-529bb" Nov 03 01:24:01.403479 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:24:01.403400 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96kxx\" (UniqueName: \"kubernetes.io/projected/096e923b-f4be-4e2b-913f-a2ecb011a0c0-kube-api-access-96kxx\") pod \"postgres-7cdbf6dc44-529bb\" (UID: \"096e923b-f4be-4e2b-913f-a2ecb011a0c0\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-529bb" Nov 03 01:24:01.407051 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:24:01.407025 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-529bb" Nov 03 01:24:01.771777 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:24:01.771749 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod096e923b_f4be_4e2b_913f_a2ecb011a0c0.slice/crio-a707c1f9b9f9e907034b34f832312322ca2eb9c0620d8f3a67dd770e093ca446 WatchSource:0}: Error finding container a707c1f9b9f9e907034b34f832312322ca2eb9c0620d8f3a67dd770e093ca446: Status 404 returned error can't find the container with id a707c1f9b9f9e907034b34f832312322ca2eb9c0620d8f3a67dd770e093ca446 Nov 03 01:24:01.773498 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:24:01.773472 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-529bb"] Nov 03 01:24:02.316379 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:24:02.316327 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-529bb" event={"ID":"096e923b-f4be-4e2b-913f-a2ecb011a0c0","Type":"ContainerStarted","Data":"a707c1f9b9f9e907034b34f832312322ca2eb9c0620d8f3a67dd770e093ca446"} Nov 03 01:24:02.351092 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:24:02.351055 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-94pkh"] Nov 03 01:24:02.353882 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:24:02.353857 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-94pkh" Nov 03 01:24:02.408525 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:24:02.408486 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 03 01:24:02.408949 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:24:02.408588 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-4cxp2\"" Nov 03 01:24:02.408949 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:24:02.408610 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 03 01:24:02.448985 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:24:02.448947 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a2288409-ad83-494d-8e5f-17cc3690265e-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-94pkh\" (UID: \"a2288409-ad83-494d-8e5f-17cc3690265e\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-94pkh" Nov 03 01:24:02.448985 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:24:02.448983 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a2288409-ad83-494d-8e5f-17cc3690265e-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-94pkh\" (UID: \"a2288409-ad83-494d-8e5f-17cc3690265e\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-94pkh" Nov 03 01:24:02.449207 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:24:02.449097 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rk87g\" (UniqueName: \"kubernetes.io/projected/a2288409-ad83-494d-8e5f-17cc3690265e-kube-api-access-rk87g\") pod \"kserve-module-controller-manager-7566cf5fc8-94pkh\" (UID: \"a2288409-ad83-494d-8e5f-17cc3690265e\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-94pkh" Nov 03 01:24:02.449250 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:24:02.449207 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a2288409-ad83-494d-8e5f-17cc3690265e-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-94pkh\" (UID: \"a2288409-ad83-494d-8e5f-17cc3690265e\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-94pkh" Nov 03 01:24:02.550612 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:24:02.550568 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rk87g\" (UniqueName: \"kubernetes.io/projected/a2288409-ad83-494d-8e5f-17cc3690265e-kube-api-access-rk87g\") pod \"kserve-module-controller-manager-7566cf5fc8-94pkh\" (UID: \"a2288409-ad83-494d-8e5f-17cc3690265e\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-94pkh" Nov 03 01:24:02.550789 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:24:02.550642 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a2288409-ad83-494d-8e5f-17cc3690265e-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-94pkh\" (UID: \"a2288409-ad83-494d-8e5f-17cc3690265e\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-94pkh" Nov 03 01:24:02.550789 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:24:02.550689 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a2288409-ad83-494d-8e5f-17cc3690265e-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-94pkh\" (UID: \"a2288409-ad83-494d-8e5f-17cc3690265e\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-94pkh" Nov 03 01:24:02.550789 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:24:02.550716 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a2288409-ad83-494d-8e5f-17cc3690265e-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-94pkh\" (UID: \"a2288409-ad83-494d-8e5f-17cc3690265e\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-94pkh" Nov 03 01:24:02.551053 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:24:02.551031 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a2288409-ad83-494d-8e5f-17cc3690265e-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-94pkh\" (UID: \"a2288409-ad83-494d-8e5f-17cc3690265e\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-94pkh" Nov 03 01:24:02.551090 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:24:02.551067 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a2288409-ad83-494d-8e5f-17cc3690265e-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-94pkh\" (UID: \"a2288409-ad83-494d-8e5f-17cc3690265e\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-94pkh" Nov 03 01:24:02.564169 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:24:02.564125 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\"" Nov 03 01:24:02.565175 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:24:02.565151 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-94pkh"] Nov 03 01:24:02.572618 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:24:02.572563 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a2288409-ad83-494d-8e5f-17cc3690265e-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-94pkh\" (UID: \"a2288409-ad83-494d-8e5f-17cc3690265e\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-94pkh" Nov 03 01:24:02.613972 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:24:02.613940 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rk87g\" (UniqueName: \"kubernetes.io/projected/a2288409-ad83-494d-8e5f-17cc3690265e-kube-api-access-rk87g\") pod \"kserve-module-controller-manager-7566cf5fc8-94pkh\" (UID: \"a2288409-ad83-494d-8e5f-17cc3690265e\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-94pkh" Nov 03 01:24:02.665945 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:24:02.665904 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-94pkh" Nov 03 01:24:03.079464 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:24:03.079431 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2288409_ad83_494d_8e5f_17cc3690265e.slice/crio-cddc04d09dce2a3bd1fb9a440070a73201a6cc4cd9d668ebde308f4c7570eb58 WatchSource:0}: Error finding container cddc04d09dce2a3bd1fb9a440070a73201a6cc4cd9d668ebde308f4c7570eb58: Status 404 returned error can't find the container with id cddc04d09dce2a3bd1fb9a440070a73201a6cc4cd9d668ebde308f4c7570eb58 Nov 03 01:24:03.080277 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:24:03.080247 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-94pkh"] Nov 03 01:24:03.321966 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:24:03.321929 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-94pkh" event={"ID":"a2288409-ad83-494d-8e5f-17cc3690265e","Type":"ContainerStarted","Data":"cddc04d09dce2a3bd1fb9a440070a73201a6cc4cd9d668ebde308f4c7570eb58"} Nov 03 01:24:06.340121 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:24:06.340077 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-94pkh" event={"ID":"a2288409-ad83-494d-8e5f-17cc3690265e","Type":"ContainerStarted","Data":"da7dfcae4d07b3488c23f0667fb593a24280e980aeae2f24563a18ba6711e95a"} Nov 03 01:24:06.340597 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:24:06.340196 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-94pkh" Nov 03 01:24:06.514336 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:24:06.514276 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-94pkh" podStartSLOduration=2.134151118 podStartE2EDuration="4.514260437s" totalImagesPullingTime="2.380109319s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:24:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:24:06.513735724 +0000 UTC m=+328.543218020" watchObservedRunningTime="2026-11-03 01:24:06.514260437 +0000 UTC m=+328.543742732" Nov 03 01:24:07.669771 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:24:07.669734 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:24:07.924397 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:24:07.924295 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:24:09.355782 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:24:09.355743 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-529bb" event={"ID":"096e923b-f4be-4e2b-913f-a2ecb011a0c0","Type":"ContainerStarted","Data":"0d91900a1e6571d493c52aaf1bda0251694259a22b5a9a684d891327f6ee7661"} Nov 03 01:24:09.356324 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:24:09.355879 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-529bb" Nov 03 01:24:09.474760 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:24:09.474705 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-529bb" podStartSLOduration=1.520719291 podStartE2EDuration="8.474690148s" totalImagesPullingTime="6.953970857s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:24:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:24:09.474134593 +0000 UTC m=+331.503616902" watchObservedRunningTime="2026-11-03 01:24:09.474690148 +0000 UTC m=+331.504172440" Nov 03 01:24:10.115663 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:24:10.115600 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:24:10.313240 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:24:10.313204 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:24:14.033229 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:24:14.033187 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 01:24:15.390974 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:24:15.390943 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-529bb" Nov 03 01:24:23.726988 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:24:23.726942 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:24:23.741626 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:24:23.741590 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6d9bdfb4cd-2sl4v"] Nov 03 01:24:23.744945 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:24:23.744918 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6d9bdfb4cd-2sl4v" Nov 03 01:24:23.799813 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:24:23.799777 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 03 01:24:23.838143 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:24:23.838104 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjmfh\" (UniqueName: \"kubernetes.io/projected/eb01fc6d-9c53-4812-8852-604413a2dec7-kube-api-access-sjmfh\") pod \"maas-controller-6d9bdfb4cd-2sl4v\" (UID: \"eb01fc6d-9c53-4812-8852-604413a2dec7\") " pod="opendatahub/maas-controller-6d9bdfb4cd-2sl4v" Nov 03 01:24:23.838336 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:24:23.838159 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb01fc6d-9c53-4812-8852-604413a2dec7-metrics-certs\") pod \"maas-controller-6d9bdfb4cd-2sl4v\" (UID: \"eb01fc6d-9c53-4812-8852-604413a2dec7\") " pod="opendatahub/maas-controller-6d9bdfb4cd-2sl4v" Nov 03 01:24:23.838430 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:24:23.838360 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/eb01fc6d-9c53-4812-8852-604413a2dec7-webhook-cert\") pod \"maas-controller-6d9bdfb4cd-2sl4v\" (UID: \"eb01fc6d-9c53-4812-8852-604413a2dec7\") " pod="opendatahub/maas-controller-6d9bdfb4cd-2sl4v" Nov 03 01:24:23.851302 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:24:23.851252 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 03 01:24:23.851302 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:24:23.851253 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-mbqp2\"" Nov 03 01:24:23.851545 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:24:23.851482 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 03 01:24:23.852446 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:24:23.852425 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6d9bdfb4cd-2sl4v"] Nov 03 01:24:23.932619 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:24:23.932586 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:24:23.938816 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:24:23.938791 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjmfh\" (UniqueName: \"kubernetes.io/projected/eb01fc6d-9c53-4812-8852-604413a2dec7-kube-api-access-sjmfh\") pod \"maas-controller-6d9bdfb4cd-2sl4v\" (UID: \"eb01fc6d-9c53-4812-8852-604413a2dec7\") " pod="opendatahub/maas-controller-6d9bdfb4cd-2sl4v" Nov 03 01:24:23.938902 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:24:23.938833 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb01fc6d-9c53-4812-8852-604413a2dec7-metrics-certs\") pod \"maas-controller-6d9bdfb4cd-2sl4v\" (UID: \"eb01fc6d-9c53-4812-8852-604413a2dec7\") " pod="opendatahub/maas-controller-6d9bdfb4cd-2sl4v" Nov 03 01:24:23.938902 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:24:23.938889 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/eb01fc6d-9c53-4812-8852-604413a2dec7-webhook-cert\") pod \"maas-controller-6d9bdfb4cd-2sl4v\" (UID: \"eb01fc6d-9c53-4812-8852-604413a2dec7\") " pod="opendatahub/maas-controller-6d9bdfb4cd-2sl4v" Nov 03 01:24:23.941424 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:24:23.941400 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/eb01fc6d-9c53-4812-8852-604413a2dec7-webhook-cert\") pod \"maas-controller-6d9bdfb4cd-2sl4v\" (UID: \"eb01fc6d-9c53-4812-8852-604413a2dec7\") " pod="opendatahub/maas-controller-6d9bdfb4cd-2sl4v" Nov 03 01:24:23.941546 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:24:23.941430 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb01fc6d-9c53-4812-8852-604413a2dec7-metrics-certs\") pod \"maas-controller-6d9bdfb4cd-2sl4v\" (UID: \"eb01fc6d-9c53-4812-8852-604413a2dec7\") " pod="opendatahub/maas-controller-6d9bdfb4cd-2sl4v" Nov 03 01:24:24.001628 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:24:24.001551 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjmfh\" (UniqueName: \"kubernetes.io/projected/eb01fc6d-9c53-4812-8852-604413a2dec7-kube-api-access-sjmfh\") pod \"maas-controller-6d9bdfb4cd-2sl4v\" (UID: \"eb01fc6d-9c53-4812-8852-604413a2dec7\") " pod="opendatahub/maas-controller-6d9bdfb4cd-2sl4v" Nov 03 01:24:24.057974 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:24:24.057915 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6d9bdfb4cd-2sl4v" Nov 03 01:24:24.143106 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:24:24.143075 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:24:24.352027 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:24:24.351986 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:24:24.354430 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:24:24.354396 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-f48966749-d28rl"] Nov 03 01:24:24.357998 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:24:24.357979 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-f48966749-d28rl" Nov 03 01:24:24.419851 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:24:24.419818 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-f48966749-d28rl"] Nov 03 01:24:24.443277 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:24:24.443233 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xl2g\" (UniqueName: \"kubernetes.io/projected/cf2bb627-ab74-4310-8fe9-a499472795f8-kube-api-access-5xl2g\") pod \"maas-controller-f48966749-d28rl\" (UID: \"cf2bb627-ab74-4310-8fe9-a499472795f8\") " pod="opendatahub/maas-controller-f48966749-d28rl" Nov 03 01:24:24.443484 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:24:24.443310 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cf2bb627-ab74-4310-8fe9-a499472795f8-webhook-cert\") pod \"maas-controller-f48966749-d28rl\" (UID: \"cf2bb627-ab74-4310-8fe9-a499472795f8\") " pod="opendatahub/maas-controller-f48966749-d28rl" Nov 03 01:24:24.443484 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:24:24.443422 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf2bb627-ab74-4310-8fe9-a499472795f8-metrics-certs\") pod \"maas-controller-f48966749-d28rl\" (UID: \"cf2bb627-ab74-4310-8fe9-a499472795f8\") " pod="opendatahub/maas-controller-f48966749-d28rl" Nov 03 01:24:24.476434 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:24:24.476395 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb01fc6d_9c53_4812_8852_604413a2dec7.slice/crio-b3738bcabe959689c98e6802febea25bbb5f03d095222ec95041e38598834b3e WatchSource:0}: Error finding container b3738bcabe959689c98e6802febea25bbb5f03d095222ec95041e38598834b3e: Status 404 returned error can't find the container with id b3738bcabe959689c98e6802febea25bbb5f03d095222ec95041e38598834b3e Nov 03 01:24:24.479140 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:24:24.479074 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6d9bdfb4cd-2sl4v"] Nov 03 01:24:24.544675 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:24:24.544635 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xl2g\" (UniqueName: \"kubernetes.io/projected/cf2bb627-ab74-4310-8fe9-a499472795f8-kube-api-access-5xl2g\") pod \"maas-controller-f48966749-d28rl\" (UID: \"cf2bb627-ab74-4310-8fe9-a499472795f8\") " pod="opendatahub/maas-controller-f48966749-d28rl" Nov 03 01:24:24.544851 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:24:24.544703 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cf2bb627-ab74-4310-8fe9-a499472795f8-webhook-cert\") pod \"maas-controller-f48966749-d28rl\" (UID: \"cf2bb627-ab74-4310-8fe9-a499472795f8\") " pod="opendatahub/maas-controller-f48966749-d28rl" Nov 03 01:24:24.544851 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:24:24.544747 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf2bb627-ab74-4310-8fe9-a499472795f8-metrics-certs\") pod \"maas-controller-f48966749-d28rl\" (UID: \"cf2bb627-ab74-4310-8fe9-a499472795f8\") " pod="opendatahub/maas-controller-f48966749-d28rl" Nov 03 01:24:24.547192 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:24:24.547162 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cf2bb627-ab74-4310-8fe9-a499472795f8-webhook-cert\") pod \"maas-controller-f48966749-d28rl\" (UID: \"cf2bb627-ab74-4310-8fe9-a499472795f8\") " pod="opendatahub/maas-controller-f48966749-d28rl" Nov 03 01:24:24.547328 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:24:24.547172 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf2bb627-ab74-4310-8fe9-a499472795f8-metrics-certs\") pod \"maas-controller-f48966749-d28rl\" (UID: \"cf2bb627-ab74-4310-8fe9-a499472795f8\") " pod="opendatahub/maas-controller-f48966749-d28rl" Nov 03 01:24:24.607789 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:24:24.607712 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xl2g\" (UniqueName: \"kubernetes.io/projected/cf2bb627-ab74-4310-8fe9-a499472795f8-kube-api-access-5xl2g\") pod \"maas-controller-f48966749-d28rl\" (UID: \"cf2bb627-ab74-4310-8fe9-a499472795f8\") " pod="opendatahub/maas-controller-f48966749-d28rl" Nov 03 01:24:24.670878 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:24:24.670840 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-f48966749-d28rl" Nov 03 01:24:25.021471 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:24:25.021351 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:24:25.042109 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:24:25.041415 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-f48966749-d28rl"] Nov 03 01:24:25.277980 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:24:25.277887 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:24:25.423877 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:24:25.423842 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-f48966749-d28rl" event={"ID":"cf2bb627-ab74-4310-8fe9-a499472795f8","Type":"ContainerStarted","Data":"de15074280b11f046fa0c8bc9f3a016ec98a2afd4311c07ebe4be3a61eb9bc12"} Nov 03 01:24:25.425030 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:24:25.425004 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6d9bdfb4cd-2sl4v" event={"ID":"eb01fc6d-9c53-4812-8852-604413a2dec7","Type":"ContainerStarted","Data":"b3738bcabe959689c98e6802febea25bbb5f03d095222ec95041e38598834b3e"} Nov 03 01:24:27.346656 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:24:27.346550 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-94pkh" Nov 03 01:24:32.054908 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:24:32.054878 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:24:32.282453 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:24:32.282409 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:24:32.456400 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:24:32.456284 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-f48966749-d28rl" event={"ID":"cf2bb627-ab74-4310-8fe9-a499472795f8","Type":"ContainerStarted","Data":"bbbd8d0e63e183d2910014d617eda5ae2147704e29526c6cc6623f0bacfbbb7c"} Nov 03 01:24:32.456569 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:24:32.456424 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-f48966749-d28rl" Nov 03 01:24:32.457937 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:24:32.457914 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6d9bdfb4cd-2sl4v" event={"ID":"eb01fc6d-9c53-4812-8852-604413a2dec7","Type":"ContainerStarted","Data":"b629427e50d7908f9d5d3291c84813dc0dc80dc0f3a74e6c94673a1a56196b95"} Nov 03 01:24:32.458054 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:24:32.458043 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6d9bdfb4cd-2sl4v" Nov 03 01:24:32.578592 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:24:32.578538 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-f48966749-d28rl" podStartSLOduration=2.042954776 podStartE2EDuration="8.578524323s" totalImagesPullingTime="6.535569547s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:24:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:24:32.578346719 +0000 UTC m=+354.607829014" watchObservedRunningTime="2026-11-03 01:24:32.578524323 +0000 UTC m=+354.608006617" Nov 03 01:24:32.700509 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:24:32.700455 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-6d9bdfb4cd-2sl4v" podStartSLOduration=2.58499387 podStartE2EDuration="9.700439666s" totalImagesPullingTime="7.115445796s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:24:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:24:32.70004204 +0000 UTC m=+354.729524336" watchObservedRunningTime="2026-11-03 01:24:32.700439666 +0000 UTC m=+354.729921960" Nov 03 01:24:35.083442 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:24:35.083406 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:24:35.345659 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:24:35.345565 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:24:37.580542 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:24:37.580501 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:24:43.465945 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:24:43.465908 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6d9bdfb4cd-2sl4v" Nov 03 01:24:43.466445 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:24:43.465998 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-f48966749-d28rl" Nov 03 01:24:43.740490 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:24:43.740325 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6d9bdfb4cd-2sl4v"] Nov 03 01:24:43.740661 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:24:43.740566 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-6d9bdfb4cd-2sl4v" podUID="eb01fc6d-9c53-4812-8852-604413a2dec7" containerName="manager" containerID="cri-o://b629427e50d7908f9d5d3291c84813dc0dc80dc0f3a74e6c94673a1a56196b95" gracePeriod=30 Nov 03 01:24:43.980799 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:24:43.980772 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6d9bdfb4cd-2sl4v" Nov 03 01:24:44.036455 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:24:44.036350 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 01:24:44.121965 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:24:44.121925 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb01fc6d-9c53-4812-8852-604413a2dec7-kube-api-access-sjmfh\" (UniqueName: \"kubernetes.io/projected/eb01fc6d-9c53-4812-8852-604413a2dec7-kube-api-access-sjmfh\") pod \"eb01fc6d-9c53-4812-8852-604413a2dec7\" (UID: \"eb01fc6d-9c53-4812-8852-604413a2dec7\") " Nov 03 01:24:44.122161 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:24:44.121973 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb01fc6d-9c53-4812-8852-604413a2dec7-webhook-cert\" (UniqueName: \"kubernetes.io/secret/eb01fc6d-9c53-4812-8852-604413a2dec7-webhook-cert\") pod \"eb01fc6d-9c53-4812-8852-604413a2dec7\" (UID: \"eb01fc6d-9c53-4812-8852-604413a2dec7\") " Nov 03 01:24:44.122161 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:24:44.122002 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb01fc6d-9c53-4812-8852-604413a2dec7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb01fc6d-9c53-4812-8852-604413a2dec7-metrics-certs\") pod \"eb01fc6d-9c53-4812-8852-604413a2dec7\" (UID: \"eb01fc6d-9c53-4812-8852-604413a2dec7\") " Nov 03 01:24:44.124338 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:24:44.124298 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb01fc6d-9c53-4812-8852-604413a2dec7-metrics-certs" pod "eb01fc6d-9c53-4812-8852-604413a2dec7" (UID: "eb01fc6d-9c53-4812-8852-604413a2dec7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:24:44.124620 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:24:44.124591 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb01fc6d-9c53-4812-8852-604413a2dec7-kube-api-access-sjmfh" pod "eb01fc6d-9c53-4812-8852-604413a2dec7" (UID: "eb01fc6d-9c53-4812-8852-604413a2dec7"). InnerVolumeSpecName "kube-api-access-sjmfh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:24:44.124724 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:24:44.124664 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb01fc6d-9c53-4812-8852-604413a2dec7-webhook-cert" pod "eb01fc6d-9c53-4812-8852-604413a2dec7" (UID: "eb01fc6d-9c53-4812-8852-604413a2dec7"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:24:44.223192 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:24:44.223156 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjmfh\" (UniqueName: \"kubernetes.io/projected/eb01fc6d-9c53-4812-8852-604413a2dec7-kube-api-access-sjmfh\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:24:44.223192 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:24:44.223188 5437 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/eb01fc6d-9c53-4812-8852-604413a2dec7-webhook-cert\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:24:44.223192 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:24:44.223201 5437 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb01fc6d-9c53-4812-8852-604413a2dec7-metrics-certs\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:24:44.508482 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:24:44.508448 5437 generic.go:356] "Generic (PLEG): container finished" podID="eb01fc6d-9c53-4812-8852-604413a2dec7" containerID="b629427e50d7908f9d5d3291c84813dc0dc80dc0f3a74e6c94673a1a56196b95" exitCode=0 Nov 03 01:24:44.508900 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:24:44.508501 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6d9bdfb4cd-2sl4v" Nov 03 01:24:44.508900 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:24:44.508533 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6d9bdfb4cd-2sl4v" event={"ID":"eb01fc6d-9c53-4812-8852-604413a2dec7","Type":"ContainerDied","Data":"b629427e50d7908f9d5d3291c84813dc0dc80dc0f3a74e6c94673a1a56196b95"} Nov 03 01:24:44.508900 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:24:44.508592 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6d9bdfb4cd-2sl4v" event={"ID":"eb01fc6d-9c53-4812-8852-604413a2dec7","Type":"ContainerDied","Data":"b3738bcabe959689c98e6802febea25bbb5f03d095222ec95041e38598834b3e"} Nov 03 01:24:44.508900 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:24:44.508621 5437 scope.go:122] "RemoveContainer" containerID="b629427e50d7908f9d5d3291c84813dc0dc80dc0f3a74e6c94673a1a56196b95" Nov 03 01:24:44.516866 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:24:44.516849 5437 scope.go:122] "RemoveContainer" containerID="b629427e50d7908f9d5d3291c84813dc0dc80dc0f3a74e6c94673a1a56196b95" Nov 03 01:24:44.517151 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:24:44.517131 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b629427e50d7908f9d5d3291c84813dc0dc80dc0f3a74e6c94673a1a56196b95\": container with ID starting with b629427e50d7908f9d5d3291c84813dc0dc80dc0f3a74e6c94673a1a56196b95 not found: ID does not exist" containerID="b629427e50d7908f9d5d3291c84813dc0dc80dc0f3a74e6c94673a1a56196b95" Nov 03 01:24:44.517210 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:24:44.517165 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b629427e50d7908f9d5d3291c84813dc0dc80dc0f3a74e6c94673a1a56196b95"} err="failed to get container status \"b629427e50d7908f9d5d3291c84813dc0dc80dc0f3a74e6c94673a1a56196b95\": rpc error: code = NotFound desc = could not find container \"b629427e50d7908f9d5d3291c84813dc0dc80dc0f3a74e6c94673a1a56196b95\": container with ID starting with b629427e50d7908f9d5d3291c84813dc0dc80dc0f3a74e6c94673a1a56196b95 not found: ID does not exist" Nov 03 01:24:44.676881 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:24:44.676848 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6d9bdfb4cd-2sl4v"] Nov 03 01:24:44.680936 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:24:44.680908 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6d9bdfb4cd-2sl4v"] Nov 03 01:24:46.086881 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:24:46.086843 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb01fc6d-9c53-4812-8852-604413a2dec7" path="/var/lib/kubelet/pods/eb01fc6d-9c53-4812-8852-604413a2dec7/volumes" Nov 03 01:24:59.943818 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:24:59.943725 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:25:00.196907 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:25:00.196810 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:25:01.291771 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:25:01.291729 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-678dcd66b5-gcrsv"] Nov 03 01:25:01.292237 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:25:01.292136 5437 cpu_manager.go:414] "containerMap: removing container" podUID="eb01fc6d-9c53-4812-8852-604413a2dec7" containerName="manager" Nov 03 01:25:01.292237 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:25:01.292150 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb01fc6d-9c53-4812-8852-604413a2dec7" containerName="manager" Nov 03 01:25:01.292237 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:25:01.292227 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb01fc6d-9c53-4812-8852-604413a2dec7" containerName="manager" Nov 03 01:25:01.301137 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:25:01.301103 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-678dcd66b5-gcrsv" Nov 03 01:25:01.349914 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:25:01.349882 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-678dcd66b5-gcrsv"] Nov 03 01:25:01.393330 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:25:01.393278 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9f4bv\" (UniqueName: \"kubernetes.io/projected/70cc650d-7318-4d94-b1f3-669d6bd64a96-kube-api-access-9f4bv\") pod \"jobset-controller-manager-678dcd66b5-gcrsv\" (UID: \"70cc650d-7318-4d94-b1f3-669d6bd64a96\") " pod="openshift-jobset-operator/jobset-controller-manager-678dcd66b5-gcrsv" Nov 03 01:25:01.393330 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:25:01.393329 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/70cc650d-7318-4d94-b1f3-669d6bd64a96-cert\") pod \"jobset-controller-manager-678dcd66b5-gcrsv\" (UID: \"70cc650d-7318-4d94-b1f3-669d6bd64a96\") " pod="openshift-jobset-operator/jobset-controller-manager-678dcd66b5-gcrsv" Nov 03 01:25:01.393595 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:25:01.393441 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70cc650d-7318-4d94-b1f3-669d6bd64a96-metrics-certs\") pod \"jobset-controller-manager-678dcd66b5-gcrsv\" (UID: \"70cc650d-7318-4d94-b1f3-669d6bd64a96\") " pod="openshift-jobset-operator/jobset-controller-manager-678dcd66b5-gcrsv" Nov 03 01:25:01.393595 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:25:01.393479 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/70cc650d-7318-4d94-b1f3-669d6bd64a96-manager-config\") pod \"jobset-controller-manager-678dcd66b5-gcrsv\" (UID: \"70cc650d-7318-4d94-b1f3-669d6bd64a96\") " pod="openshift-jobset-operator/jobset-controller-manager-678dcd66b5-gcrsv" Nov 03 01:25:01.400608 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:25:01.400575 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\"" Nov 03 01:25:01.400608 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:25:01.400599 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Nov 03 01:25:01.400842 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:25:01.400671 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\"" Nov 03 01:25:01.400842 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:25:01.400713 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\"" Nov 03 01:25:01.406845 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:25:01.406824 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Nov 03 01:25:01.406942 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:25:01.406846 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-cpr8n\"" Nov 03 01:25:01.493875 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:25:01.493837 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/70cc650d-7318-4d94-b1f3-669d6bd64a96-cert\") pod \"jobset-controller-manager-678dcd66b5-gcrsv\" (UID: \"70cc650d-7318-4d94-b1f3-669d6bd64a96\") " pod="openshift-jobset-operator/jobset-controller-manager-678dcd66b5-gcrsv" Nov 03 01:25:01.494092 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:25:01.493898 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70cc650d-7318-4d94-b1f3-669d6bd64a96-metrics-certs\") pod \"jobset-controller-manager-678dcd66b5-gcrsv\" (UID: \"70cc650d-7318-4d94-b1f3-669d6bd64a96\") " pod="openshift-jobset-operator/jobset-controller-manager-678dcd66b5-gcrsv" Nov 03 01:25:01.494092 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:25:01.493938 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/70cc650d-7318-4d94-b1f3-669d6bd64a96-manager-config\") pod \"jobset-controller-manager-678dcd66b5-gcrsv\" (UID: \"70cc650d-7318-4d94-b1f3-669d6bd64a96\") " pod="openshift-jobset-operator/jobset-controller-manager-678dcd66b5-gcrsv" Nov 03 01:25:01.494092 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:25:01.494003 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9f4bv\" (UniqueName: \"kubernetes.io/projected/70cc650d-7318-4d94-b1f3-669d6bd64a96-kube-api-access-9f4bv\") pod \"jobset-controller-manager-678dcd66b5-gcrsv\" (UID: \"70cc650d-7318-4d94-b1f3-669d6bd64a96\") " pod="openshift-jobset-operator/jobset-controller-manager-678dcd66b5-gcrsv" Nov 03 01:25:01.494707 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:25:01.494682 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/70cc650d-7318-4d94-b1f3-669d6bd64a96-manager-config\") pod \"jobset-controller-manager-678dcd66b5-gcrsv\" (UID: \"70cc650d-7318-4d94-b1f3-669d6bd64a96\") " pod="openshift-jobset-operator/jobset-controller-manager-678dcd66b5-gcrsv" Nov 03 01:25:01.496614 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:25:01.496587 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70cc650d-7318-4d94-b1f3-669d6bd64a96-metrics-certs\") pod \"jobset-controller-manager-678dcd66b5-gcrsv\" (UID: \"70cc650d-7318-4d94-b1f3-669d6bd64a96\") " pod="openshift-jobset-operator/jobset-controller-manager-678dcd66b5-gcrsv" Nov 03 01:25:01.496707 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:25:01.496596 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/70cc650d-7318-4d94-b1f3-669d6bd64a96-cert\") pod \"jobset-controller-manager-678dcd66b5-gcrsv\" (UID: \"70cc650d-7318-4d94-b1f3-669d6bd64a96\") " pod="openshift-jobset-operator/jobset-controller-manager-678dcd66b5-gcrsv" Nov 03 01:25:01.556732 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:25:01.556693 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9f4bv\" (UniqueName: \"kubernetes.io/projected/70cc650d-7318-4d94-b1f3-669d6bd64a96-kube-api-access-9f4bv\") pod \"jobset-controller-manager-678dcd66b5-gcrsv\" (UID: \"70cc650d-7318-4d94-b1f3-669d6bd64a96\") " pod="openshift-jobset-operator/jobset-controller-manager-678dcd66b5-gcrsv" Nov 03 01:25:01.615166 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:25:01.615126 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-678dcd66b5-gcrsv" Nov 03 01:25:01.988297 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:25:01.988269 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70cc650d_7318_4d94_b1f3_669d6bd64a96.slice/crio-254cd235130cfe503f562f755ad5be1a24ff5d1da59e422a441cd3c21055b302 WatchSource:0}: Error finding container 254cd235130cfe503f562f755ad5be1a24ff5d1da59e422a441cd3c21055b302: Status 404 returned error can't find the container with id 254cd235130cfe503f562f755ad5be1a24ff5d1da59e422a441cd3c21055b302 Nov 03 01:25:01.990338 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:25:01.990314 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-678dcd66b5-gcrsv"] Nov 03 01:25:02.584251 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:25:02.584216 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-678dcd66b5-gcrsv" event={"ID":"70cc650d-7318-4d94-b1f3-669d6bd64a96","Type":"ContainerStarted","Data":"254cd235130cfe503f562f755ad5be1a24ff5d1da59e422a441cd3c21055b302"} Nov 03 01:25:06.335298 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:25:06.335256 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:25:06.586781 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:25:06.586690 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:25:07.607572 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:25:07.607533 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-678dcd66b5-gcrsv" event={"ID":"70cc650d-7318-4d94-b1f3-669d6bd64a96","Type":"ContainerStarted","Data":"c1265781b983b47974f89cedd9fe6e932ad568b98ee1be4fcbc2ae1f8f7de3d0"} Nov 03 01:25:07.607998 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:25:07.607627 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-678dcd66b5-gcrsv" Nov 03 01:25:07.728032 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:25:07.727974 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-678dcd66b5-gcrsv" podStartSLOduration=1.602433391 podStartE2EDuration="6.727960741s" totalImagesPullingTime="5.12552735s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:25:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:25:07.727196731 +0000 UTC m=+389.756679025" watchObservedRunningTime="2026-11-03 01:25:07.727960741 +0000 UTC m=+389.757443036" Nov 03 01:25:08.449260 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:25:08.449221 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:25:08.668825 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:25:08.668791 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:25:08.880312 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:25:08.880266 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:25:09.084517 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:25:09.084482 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:25:09.782517 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:25:09.782472 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:25:09.984359 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:25:09.984309 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:25:10.184250 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:25:10.184202 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:25:10.427656 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:25:10.427612 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:25:10.637928 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:25:10.637893 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:25:10.842617 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:25:10.842579 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:25:11.710916 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:25:11.710866 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-85chp"] Nov 03 01:25:11.711130 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:25:11.711091 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-85chp" podUID="c1b1cb38-6cd4-41bc-8c11-97951ddbe4af" containerName="discovery" containerID="cri-o://a99adae7582e8f0bd8fdd54ef7c8e56fe99ef3188b340cd06e169cd24247ce9a" gracePeriod=30 Nov 03 01:25:11.956550 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:25:11.956523 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-85chp" Nov 03 01:25:11.988719 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:25:11.988650 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1b1cb38-6cd4-41bc-8c11-97951ddbe4af-cacerts\" (UniqueName: \"kubernetes.io/secret/c1b1cb38-6cd4-41bc-8c11-97951ddbe4af-cacerts\") pod \"c1b1cb38-6cd4-41bc-8c11-97951ddbe4af\" (UID: \"c1b1cb38-6cd4-41bc-8c11-97951ddbe4af\") " Nov 03 01:25:11.988719 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:25:11.988696 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1b1cb38-6cd4-41bc-8c11-97951ddbe4af-kube-api-access-l299p\" (UniqueName: \"kubernetes.io/projected/c1b1cb38-6cd4-41bc-8c11-97951ddbe4af-kube-api-access-l299p\") pod \"c1b1cb38-6cd4-41bc-8c11-97951ddbe4af\" (UID: \"c1b1cb38-6cd4-41bc-8c11-97951ddbe4af\") " Nov 03 01:25:11.988938 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:25:11.988738 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1b1cb38-6cd4-41bc-8c11-97951ddbe4af-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1b1cb38-6cd4-41bc-8c11-97951ddbe4af-local-certs\") pod \"c1b1cb38-6cd4-41bc-8c11-97951ddbe4af\" (UID: \"c1b1cb38-6cd4-41bc-8c11-97951ddbe4af\") " Nov 03 01:25:11.988938 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:25:11.988904 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1b1cb38-6cd4-41bc-8c11-97951ddbe4af-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c1b1cb38-6cd4-41bc-8c11-97951ddbe4af-istio-csr-dns-cert\") pod \"c1b1cb38-6cd4-41bc-8c11-97951ddbe4af\" (UID: \"c1b1cb38-6cd4-41bc-8c11-97951ddbe4af\") " Nov 03 01:25:11.989041 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:25:11.988960 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1b1cb38-6cd4-41bc-8c11-97951ddbe4af-istio-token\" (UniqueName: \"kubernetes.io/projected/c1b1cb38-6cd4-41bc-8c11-97951ddbe4af-istio-token\") pod \"c1b1cb38-6cd4-41bc-8c11-97951ddbe4af\" (UID: \"c1b1cb38-6cd4-41bc-8c11-97951ddbe4af\") " Nov 03 01:25:11.989041 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:25:11.989000 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1b1cb38-6cd4-41bc-8c11-97951ddbe4af-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c1b1cb38-6cd4-41bc-8c11-97951ddbe4af-istio-kubeconfig\") pod \"c1b1cb38-6cd4-41bc-8c11-97951ddbe4af\" (UID: \"c1b1cb38-6cd4-41bc-8c11-97951ddbe4af\") " Nov 03 01:25:11.989041 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:25:11.989036 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1b1cb38-6cd4-41bc-8c11-97951ddbe4af-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c1b1cb38-6cd4-41bc-8c11-97951ddbe4af-istio-csr-ca-configmap\") pod \"c1b1cb38-6cd4-41bc-8c11-97951ddbe4af\" (UID: \"c1b1cb38-6cd4-41bc-8c11-97951ddbe4af\") " Nov 03 01:25:11.989765 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:25:11.989714 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1b1cb38-6cd4-41bc-8c11-97951ddbe4af-istio-csr-ca-configmap" pod "c1b1cb38-6cd4-41bc-8c11-97951ddbe4af" (UID: "c1b1cb38-6cd4-41bc-8c11-97951ddbe4af"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:25:11.991174 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:25:11.991143 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1b1cb38-6cd4-41bc-8c11-97951ddbe4af-cacerts" pod "c1b1cb38-6cd4-41bc-8c11-97951ddbe4af" (UID: "c1b1cb38-6cd4-41bc-8c11-97951ddbe4af"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:25:11.991288 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:25:11.991230 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1b1cb38-6cd4-41bc-8c11-97951ddbe4af-kube-api-access-l299p" pod "c1b1cb38-6cd4-41bc-8c11-97951ddbe4af" (UID: "c1b1cb38-6cd4-41bc-8c11-97951ddbe4af"). InnerVolumeSpecName "kube-api-access-l299p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:25:11.991398 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:25:11.991352 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1b1cb38-6cd4-41bc-8c11-97951ddbe4af-local-certs" pod "c1b1cb38-6cd4-41bc-8c11-97951ddbe4af" (UID: "c1b1cb38-6cd4-41bc-8c11-97951ddbe4af"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:25:11.991561 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:25:11.991532 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1b1cb38-6cd4-41bc-8c11-97951ddbe4af-istio-csr-dns-cert" pod "c1b1cb38-6cd4-41bc-8c11-97951ddbe4af" (UID: "c1b1cb38-6cd4-41bc-8c11-97951ddbe4af"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:25:11.991669 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:25:11.991558 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1b1cb38-6cd4-41bc-8c11-97951ddbe4af-istio-token" pod "c1b1cb38-6cd4-41bc-8c11-97951ddbe4af" (UID: "c1b1cb38-6cd4-41bc-8c11-97951ddbe4af"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:25:11.991669 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:25:11.991558 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1b1cb38-6cd4-41bc-8c11-97951ddbe4af-istio-kubeconfig" pod "c1b1cb38-6cd4-41bc-8c11-97951ddbe4af" (UID: "c1b1cb38-6cd4-41bc-8c11-97951ddbe4af"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:25:12.090374 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:25:12.090330 5437 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c1b1cb38-6cd4-41bc-8c11-97951ddbe4af-istio-csr-dns-cert\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:25:12.090374 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:25:12.090358 5437 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1b1cb38-6cd4-41bc-8c11-97951ddbe4af-istio-token\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:25:12.090374 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:25:12.090379 5437 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c1b1cb38-6cd4-41bc-8c11-97951ddbe4af-istio-kubeconfig\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:25:12.090744 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:25:12.090389 5437 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c1b1cb38-6cd4-41bc-8c11-97951ddbe4af-istio-csr-ca-configmap\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:25:12.090744 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:25:12.090401 5437 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c1b1cb38-6cd4-41bc-8c11-97951ddbe4af-cacerts\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:25:12.090744 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:25:12.090409 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l299p\" (UniqueName: \"kubernetes.io/projected/c1b1cb38-6cd4-41bc-8c11-97951ddbe4af-kube-api-access-l299p\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:25:12.090744 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:25:12.090417 5437 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1b1cb38-6cd4-41bc-8c11-97951ddbe4af-local-certs\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:25:12.136334 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:25:12.136302 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:25:12.628993 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:25:12.628959 5437 generic.go:356] "Generic (PLEG): container finished" podID="c1b1cb38-6cd4-41bc-8c11-97951ddbe4af" containerID="a99adae7582e8f0bd8fdd54ef7c8e56fe99ef3188b340cd06e169cd24247ce9a" exitCode=0 Nov 03 01:25:12.629166 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:25:12.629003 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-85chp" event={"ID":"c1b1cb38-6cd4-41bc-8c11-97951ddbe4af","Type":"ContainerDied","Data":"a99adae7582e8f0bd8fdd54ef7c8e56fe99ef3188b340cd06e169cd24247ce9a"} Nov 03 01:25:12.629166 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:25:12.629015 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-85chp" Nov 03 01:25:12.629166 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:25:12.629056 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-85chp" event={"ID":"c1b1cb38-6cd4-41bc-8c11-97951ddbe4af","Type":"ContainerDied","Data":"1b5779f226d83f7e6f651d86bc1cf65cb7c522f713d907a4882426f33db46207"} Nov 03 01:25:12.629166 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:25:12.629085 5437 scope.go:122] "RemoveContainer" containerID="a99adae7582e8f0bd8fdd54ef7c8e56fe99ef3188b340cd06e169cd24247ce9a" Nov 03 01:25:12.637259 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:25:12.637237 5437 scope.go:122] "RemoveContainer" containerID="a99adae7582e8f0bd8fdd54ef7c8e56fe99ef3188b340cd06e169cd24247ce9a" Nov 03 01:25:12.637547 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:25:12.637526 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a99adae7582e8f0bd8fdd54ef7c8e56fe99ef3188b340cd06e169cd24247ce9a\": container with ID starting with a99adae7582e8f0bd8fdd54ef7c8e56fe99ef3188b340cd06e169cd24247ce9a not found: ID does not exist" containerID="a99adae7582e8f0bd8fdd54ef7c8e56fe99ef3188b340cd06e169cd24247ce9a" Nov 03 01:25:12.637625 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:25:12.637555 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a99adae7582e8f0bd8fdd54ef7c8e56fe99ef3188b340cd06e169cd24247ce9a"} err="failed to get container status \"a99adae7582e8f0bd8fdd54ef7c8e56fe99ef3188b340cd06e169cd24247ce9a\": rpc error: code = NotFound desc = could not find container \"a99adae7582e8f0bd8fdd54ef7c8e56fe99ef3188b340cd06e169cd24247ce9a\": container with ID starting with a99adae7582e8f0bd8fdd54ef7c8e56fe99ef3188b340cd06e169cd24247ce9a not found: ID does not exist" Nov 03 01:25:12.849912 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:25:12.849875 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-85chp"] Nov 03 01:25:12.901306 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:25:12.901230 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-85chp"] Nov 03 01:25:14.032559 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:25:14.032523 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 01:25:14.087302 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:25:14.087265 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1b1cb38-6cd4-41bc-8c11-97951ddbe4af" path="/var/lib/kubelet/pods/c1b1cb38-6cd4-41bc-8c11-97951ddbe4af/volumes" Nov 03 01:25:16.925010 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:25:16.924972 5437 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-85chp container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.18:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 01:25:16.925436 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:25:16.925044 5437 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-85chp" podUID="c1b1cb38-6cd4-41bc-8c11-97951ddbe4af" containerName="discovery" probeResult="failure" output="Get \"http://10.134.0.18:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 01:25:17.550110 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:25:17.550057 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:25:17.822512 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:25:17.822421 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:25:18.026035 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:25:18.025974 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:25:18.227724 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:25:18.227624 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:25:18.433633 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:25:18.433589 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:25:18.617294 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:25:18.617263 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-678dcd66b5-gcrsv" Nov 03 01:25:18.637654 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:25:18.637622 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:25:18.842761 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:25:18.842722 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:25:24.342054 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:25:24.342011 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:25:24.726059 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:25:24.725961 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:25:24.812447 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:25:24.812411 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6b6d569d44-rm7ch"] Nov 03 01:25:24.812766 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:25:24.812752 5437 cpu_manager.go:414] "containerMap: removing container" podUID="c1b1cb38-6cd4-41bc-8c11-97951ddbe4af" containerName="discovery" Nov 03 01:25:24.812829 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:25:24.812767 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1b1cb38-6cd4-41bc-8c11-97951ddbe4af" containerName="discovery" Nov 03 01:25:24.812865 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:25:24.812849 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1b1cb38-6cd4-41bc-8c11-97951ddbe4af" containerName="discovery" Nov 03 01:25:24.815816 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:25:24.815797 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6b6d569d44-rm7ch" Nov 03 01:25:24.903647 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:25:24.903586 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e301e8c2-2042-4641-a416-e26c636c2dfd-maas-api-tls\") pod \"maas-api-6b6d569d44-rm7ch\" (UID: \"e301e8c2-2042-4641-a416-e26c636c2dfd\") " pod="odh-ai-gateway-infra/maas-api-6b6d569d44-rm7ch" Nov 03 01:25:24.903647 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:25:24.903651 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9c8q\" (UniqueName: \"kubernetes.io/projected/e301e8c2-2042-4641-a416-e26c636c2dfd-kube-api-access-z9c8q\") pod \"maas-api-6b6d569d44-rm7ch\" (UID: \"e301e8c2-2042-4641-a416-e26c636c2dfd\") " pod="odh-ai-gateway-infra/maas-api-6b6d569d44-rm7ch" Nov 03 01:25:24.903896 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:25:24.903785 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e301e8c2-2042-4641-a416-e26c636c2dfd-metrics-certs\") pod \"maas-api-6b6d569d44-rm7ch\" (UID: \"e301e8c2-2042-4641-a416-e26c636c2dfd\") " pod="odh-ai-gateway-infra/maas-api-6b6d569d44-rm7ch" Nov 03 01:25:24.903896 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:25:24.903839 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e301e8c2-2042-4641-a416-e26c636c2dfd-openshift-service-ca\") pod \"maas-api-6b6d569d44-rm7ch\" (UID: \"e301e8c2-2042-4641-a416-e26c636c2dfd\") " pod="odh-ai-gateway-infra/maas-api-6b6d569d44-rm7ch" Nov 03 01:25:24.922451 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:25:24.922418 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 01:25:24.922451 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:25:24.922417 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Nov 03 01:25:24.922451 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:25:24.922419 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-sxqdt\"" Nov 03 01:25:24.923144 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:25:24.923123 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6b6d569d44-rm7ch"] Nov 03 01:25:25.004329 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:25:25.004246 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e301e8c2-2042-4641-a416-e26c636c2dfd-maas-api-tls\") pod \"maas-api-6b6d569d44-rm7ch\" (UID: \"e301e8c2-2042-4641-a416-e26c636c2dfd\") " pod="odh-ai-gateway-infra/maas-api-6b6d569d44-rm7ch" Nov 03 01:25:25.004329 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:25:25.004285 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9c8q\" (UniqueName: \"kubernetes.io/projected/e301e8c2-2042-4641-a416-e26c636c2dfd-kube-api-access-z9c8q\") pod \"maas-api-6b6d569d44-rm7ch\" (UID: \"e301e8c2-2042-4641-a416-e26c636c2dfd\") " pod="odh-ai-gateway-infra/maas-api-6b6d569d44-rm7ch" Nov 03 01:25:25.004329 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:25:25.004313 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e301e8c2-2042-4641-a416-e26c636c2dfd-metrics-certs\") pod \"maas-api-6b6d569d44-rm7ch\" (UID: \"e301e8c2-2042-4641-a416-e26c636c2dfd\") " pod="odh-ai-gateway-infra/maas-api-6b6d569d44-rm7ch" Nov 03 01:25:25.004653 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:25:25.004342 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e301e8c2-2042-4641-a416-e26c636c2dfd-openshift-service-ca\") pod \"maas-api-6b6d569d44-rm7ch\" (UID: \"e301e8c2-2042-4641-a416-e26c636c2dfd\") " pod="odh-ai-gateway-infra/maas-api-6b6d569d44-rm7ch" Nov 03 01:25:25.005007 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:25:25.004974 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e301e8c2-2042-4641-a416-e26c636c2dfd-openshift-service-ca\") pod \"maas-api-6b6d569d44-rm7ch\" (UID: \"e301e8c2-2042-4641-a416-e26c636c2dfd\") " pod="odh-ai-gateway-infra/maas-api-6b6d569d44-rm7ch" Nov 03 01:25:25.006930 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:25:25.006903 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e301e8c2-2042-4641-a416-e26c636c2dfd-metrics-certs\") pod \"maas-api-6b6d569d44-rm7ch\" (UID: \"e301e8c2-2042-4641-a416-e26c636c2dfd\") " pod="odh-ai-gateway-infra/maas-api-6b6d569d44-rm7ch" Nov 03 01:25:25.007032 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:25:25.006903 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e301e8c2-2042-4641-a416-e26c636c2dfd-maas-api-tls\") pod \"maas-api-6b6d569d44-rm7ch\" (UID: \"e301e8c2-2042-4641-a416-e26c636c2dfd\") " pod="odh-ai-gateway-infra/maas-api-6b6d569d44-rm7ch" Nov 03 01:25:25.067305 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:25:25.067271 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9c8q\" (UniqueName: \"kubernetes.io/projected/e301e8c2-2042-4641-a416-e26c636c2dfd-kube-api-access-z9c8q\") pod \"maas-api-6b6d569d44-rm7ch\" (UID: \"e301e8c2-2042-4641-a416-e26c636c2dfd\") " pod="odh-ai-gateway-infra/maas-api-6b6d569d44-rm7ch" Nov 03 01:25:25.128547 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:25:25.128508 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6b6d569d44-rm7ch" Nov 03 01:25:25.496597 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:25:25.496561 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode301e8c2_2042_4641_a416_e26c636c2dfd.slice/crio-505a023e5782077e5953c01b87a9f8dd9914f48b217cf24f16cafffc45319aa9 WatchSource:0}: Error finding container 505a023e5782077e5953c01b87a9f8dd9914f48b217cf24f16cafffc45319aa9: Status 404 returned error can't find the container with id 505a023e5782077e5953c01b87a9f8dd9914f48b217cf24f16cafffc45319aa9 Nov 03 01:25:25.500334 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:25:25.500296 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6b6d569d44-rm7ch"] Nov 03 01:25:25.686037 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:25:25.685999 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6b6d569d44-rm7ch" event={"ID":"e301e8c2-2042-4641-a416-e26c636c2dfd","Type":"ContainerStarted","Data":"505a023e5782077e5953c01b87a9f8dd9914f48b217cf24f16cafffc45319aa9"} Nov 03 01:25:28.703600 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:25:28.703563 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6b6d569d44-rm7ch" event={"ID":"e301e8c2-2042-4641-a416-e26c636c2dfd","Type":"ContainerStarted","Data":"9e0f91fdc0e28715bc88d22d937818a08b3a2a74ef18c7f12fe9f0fe1961a71e"} Nov 03 01:25:28.704034 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:25:28.703647 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6b6d569d44-rm7ch" Nov 03 01:25:28.824243 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:25:28.824187 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6b6d569d44-rm7ch" podStartSLOduration=2.693517938 podStartE2EDuration="4.824172233s" totalImagesPullingTime="2.130654295s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:25:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:25:28.823779282 +0000 UTC m=+410.853261576" watchObservedRunningTime="2026-11-03 01:25:28.824172233 +0000 UTC m=+410.853654532" Nov 03 01:25:34.714020 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:25:34.713983 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6b6d569d44-rm7ch" Nov 03 01:25:37.923150 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:25:37.923115 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:25:41.958908 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:25:41.958867 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:25:44.033966 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:25:44.033927 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 01:26:14.032975 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:26:14.032933 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 01:26:38.296538 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:26:38.296481 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:26:38.502747 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:26:38.502713 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:26:44.034799 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:26:44.034760 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 01:26:46.613585 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:26:46.613550 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:26:46.813581 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:26:46.813547 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:27:04.116019 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:04.115982 5437 generic.go:356] "Generic (PLEG): container finished" podID="cf2bb627-ab74-4310-8fe9-a499472795f8" containerID="bbbd8d0e63e183d2910014d617eda5ae2147704e29526c6cc6623f0bacfbbb7c" exitCode=1 Nov 03 01:27:04.116466 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:04.116052 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-f48966749-d28rl" event={"ID":"cf2bb627-ab74-4310-8fe9-a499472795f8","Type":"ContainerDied","Data":"bbbd8d0e63e183d2910014d617eda5ae2147704e29526c6cc6623f0bacfbbb7c"} Nov 03 01:27:04.116466 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:04.116421 5437 scope.go:122] "RemoveContainer" containerID="bbbd8d0e63e183d2910014d617eda5ae2147704e29526c6cc6623f0bacfbbb7c" Nov 03 01:27:04.419823 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:04.419788 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-tvq28"] Nov 03 01:27:04.423200 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:04.423179 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d8dddb676-tvq28" Nov 03 01:27:04.470683 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:04.470649 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:27:04.477563 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:04.477533 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-tvq28"] Nov 03 01:27:04.477792 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:04.477771 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\"" Nov 03 01:27:04.477851 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:04.477801 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Nov 03 01:27:04.477851 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:04.477844 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-hm6qw\"" Nov 03 01:27:04.555395 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:04.555327 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/564648fe-209b-42e7-8d01-f922683d0a3a-metrics-certs\") pod \"ai-gateway-controller-5d8dddb676-tvq28\" (UID: \"564648fe-209b-42e7-8d01-f922683d0a3a\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-tvq28" Nov 03 01:27:04.555588 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:04.555454 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bbkj\" (UniqueName: \"kubernetes.io/projected/564648fe-209b-42e7-8d01-f922683d0a3a-kube-api-access-2bbkj\") pod \"ai-gateway-controller-5d8dddb676-tvq28\" (UID: \"564648fe-209b-42e7-8d01-f922683d0a3a\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-tvq28" Nov 03 01:27:04.655960 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:04.655912 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/564648fe-209b-42e7-8d01-f922683d0a3a-metrics-certs\") pod \"ai-gateway-controller-5d8dddb676-tvq28\" (UID: \"564648fe-209b-42e7-8d01-f922683d0a3a\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-tvq28" Nov 03 01:27:04.655960 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:04.655969 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bbkj\" (UniqueName: \"kubernetes.io/projected/564648fe-209b-42e7-8d01-f922683d0a3a-kube-api-access-2bbkj\") pod \"ai-gateway-controller-5d8dddb676-tvq28\" (UID: \"564648fe-209b-42e7-8d01-f922683d0a3a\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-tvq28" Nov 03 01:27:04.658375 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:04.658331 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/564648fe-209b-42e7-8d01-f922683d0a3a-metrics-certs\") pod \"ai-gateway-controller-5d8dddb676-tvq28\" (UID: \"564648fe-209b-42e7-8d01-f922683d0a3a\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-tvq28" Nov 03 01:27:04.671547 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:04.671477 5437 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="opendatahub/maas-controller-f48966749-d28rl" Nov 03 01:27:04.681341 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:04.681305 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:27:04.719212 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:04.719178 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bbkj\" (UniqueName: \"kubernetes.io/projected/564648fe-209b-42e7-8d01-f922683d0a3a-kube-api-access-2bbkj\") pod \"ai-gateway-controller-5d8dddb676-tvq28\" (UID: \"564648fe-209b-42e7-8d01-f922683d0a3a\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-tvq28" Nov 03 01:27:04.735754 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:04.735724 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d8dddb676-tvq28" Nov 03 01:27:05.101467 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:27:05.101437 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod564648fe_209b_42e7_8d01_f922683d0a3a.slice/crio-844248599bca674ffa68234088b12500d21d3ced3fb2cd66870705d20bcd6373 WatchSource:0}: Error finding container 844248599bca674ffa68234088b12500d21d3ced3fb2cd66870705d20bcd6373: Status 404 returned error can't find the container with id 844248599bca674ffa68234088b12500d21d3ced3fb2cd66870705d20bcd6373 Nov 03 01:27:05.103142 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:05.103114 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-tvq28"] Nov 03 01:27:05.121096 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:05.121067 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d8dddb676-tvq28" event={"ID":"564648fe-209b-42e7-8d01-f922683d0a3a","Type":"ContainerStarted","Data":"844248599bca674ffa68234088b12500d21d3ced3fb2cd66870705d20bcd6373"} Nov 03 01:27:06.128031 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:06.127992 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-f48966749-d28rl" event={"ID":"cf2bb627-ab74-4310-8fe9-a499472795f8","Type":"ContainerStarted","Data":"1de541d4a3f5d0328ddcbb85a609e23d4abff77a6803a217c95378692dbc393d"} Nov 03 01:27:06.128604 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:06.128068 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-f48966749-d28rl" Nov 03 01:27:08.138419 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:08.138355 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d8dddb676-tvq28" event={"ID":"564648fe-209b-42e7-8d01-f922683d0a3a","Type":"ContainerStarted","Data":"914e537e8463cca2312a48a5728d260c5e725521600e236cd2ab52e27f313ea0"} Nov 03 01:27:08.138922 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:08.138500 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-5d8dddb676-tvq28" Nov 03 01:27:08.259093 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:08.259026 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-5d8dddb676-tvq28" podStartSLOduration=1.969203363 podStartE2EDuration="4.259009377s" totalImagesPullingTime="2.289806014s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:27:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:27:08.258044039 +0000 UTC m=+510.287526330" watchObservedRunningTime="2026-11-03 01:27:08.259009377 +0000 UTC m=+510.288491673" Nov 03 01:27:10.302005 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:10.301965 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:27:10.556556 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:10.556468 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:27:10.655665 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:10.655635 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-jltq6"] Nov 03 01:27:10.659135 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:10.659109 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-jltq6" Nov 03 01:27:10.749207 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:10.749168 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-jltq6"] Nov 03 01:27:10.758309 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:10.758283 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Nov 03 01:27:10.758427 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:10.758292 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-8fd52\"" Nov 03 01:27:10.805464 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:10.805413 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-ghxht"] Nov 03 01:27:10.808783 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:10.808725 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-ghxht" Nov 03 01:27:10.812625 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:10.812604 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdbx2\" (UniqueName: \"kubernetes.io/projected/1748acc6-ac8d-4a30-ba58-70ade1bcbd0f-kube-api-access-sdbx2\") pod \"payload-pre-processing-7b98bb678c-jltq6\" (UID: \"1748acc6-ac8d-4a30-ba58-70ade1bcbd0f\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-jltq6" Nov 03 01:27:10.812736 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:10.812716 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1748acc6-ac8d-4a30-ba58-70ade1bcbd0f-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-jltq6\" (UID: \"1748acc6-ac8d-4a30-ba58-70ade1bcbd0f\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-jltq6" Nov 03 01:27:10.863589 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:10.863550 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-ghxht"] Nov 03 01:27:10.914057 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:10.914026 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1748acc6-ac8d-4a30-ba58-70ade1bcbd0f-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-jltq6\" (UID: \"1748acc6-ac8d-4a30-ba58-70ade1bcbd0f\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-jltq6" Nov 03 01:27:10.914219 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:10.914065 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdbx2\" (UniqueName: \"kubernetes.io/projected/1748acc6-ac8d-4a30-ba58-70ade1bcbd0f-kube-api-access-sdbx2\") pod \"payload-pre-processing-7b98bb678c-jltq6\" (UID: \"1748acc6-ac8d-4a30-ba58-70ade1bcbd0f\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-jltq6" Nov 03 01:27:10.914219 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:10.914105 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cm99\" (UniqueName: \"kubernetes.io/projected/c4e1a934-d917-417b-aea3-d432d2cff990-kube-api-access-9cm99\") pod \"payload-processing-55fbbbf9dc-ghxht\" (UID: \"c4e1a934-d917-417b-aea3-d432d2cff990\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-ghxht" Nov 03 01:27:10.914219 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:10.914141 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4e1a934-d917-417b-aea3-d432d2cff990-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-ghxht\" (UID: \"c4e1a934-d917-417b-aea3-d432d2cff990\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-ghxht" Nov 03 01:27:10.914789 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:10.914766 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1748acc6-ac8d-4a30-ba58-70ade1bcbd0f-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-jltq6\" (UID: \"1748acc6-ac8d-4a30-ba58-70ade1bcbd0f\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-jltq6" Nov 03 01:27:10.977652 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:10.977607 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdbx2\" (UniqueName: \"kubernetes.io/projected/1748acc6-ac8d-4a30-ba58-70ade1bcbd0f-kube-api-access-sdbx2\") pod \"payload-pre-processing-7b98bb678c-jltq6\" (UID: \"1748acc6-ac8d-4a30-ba58-70ade1bcbd0f\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-jltq6" Nov 03 01:27:11.014661 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:11.014615 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cm99\" (UniqueName: \"kubernetes.io/projected/c4e1a934-d917-417b-aea3-d432d2cff990-kube-api-access-9cm99\") pod \"payload-processing-55fbbbf9dc-ghxht\" (UID: \"c4e1a934-d917-417b-aea3-d432d2cff990\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-ghxht" Nov 03 01:27:11.014834 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:11.014678 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4e1a934-d917-417b-aea3-d432d2cff990-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-ghxht\" (UID: \"c4e1a934-d917-417b-aea3-d432d2cff990\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-ghxht" Nov 03 01:27:11.015337 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:11.015313 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4e1a934-d917-417b-aea3-d432d2cff990-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-ghxht\" (UID: \"c4e1a934-d917-417b-aea3-d432d2cff990\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-ghxht" Nov 03 01:27:11.077472 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:11.077388 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cm99\" (UniqueName: \"kubernetes.io/projected/c4e1a934-d917-417b-aea3-d432d2cff990-kube-api-access-9cm99\") pod \"payload-processing-55fbbbf9dc-ghxht\" (UID: \"c4e1a934-d917-417b-aea3-d432d2cff990\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-ghxht" Nov 03 01:27:11.121728 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:11.121689 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-ghxht" Nov 03 01:27:11.271765 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:11.271717 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-jltq6" Nov 03 01:27:11.491658 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:27:11.491621 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4e1a934_d917_417b_aea3_d432d2cff990.slice/crio-6965aaf93fd6b13fa4c3cda4b0e13b975c5aa6bae07227a4b21d01c4ddc628aa WatchSource:0}: Error finding container 6965aaf93fd6b13fa4c3cda4b0e13b975c5aa6bae07227a4b21d01c4ddc628aa: Status 404 returned error can't find the container with id 6965aaf93fd6b13fa4c3cda4b0e13b975c5aa6bae07227a4b21d01c4ddc628aa Nov 03 01:27:11.493542 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:11.493518 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-ghxht"] Nov 03 01:27:11.638252 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:27:11.638216 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1748acc6_ac8d_4a30_ba58_70ade1bcbd0f.slice/crio-2f046892450de428d6cbbfdaebb6ff5b88103ba4ee4748f4b2da5102ec3c592b WatchSource:0}: Error finding container 2f046892450de428d6cbbfdaebb6ff5b88103ba4ee4748f4b2da5102ec3c592b: Status 404 returned error can't find the container with id 2f046892450de428d6cbbfdaebb6ff5b88103ba4ee4748f4b2da5102ec3c592b Nov 03 01:27:11.639402 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:11.639324 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-jltq6"] Nov 03 01:27:12.156660 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:12.156625 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-ghxht" event={"ID":"c4e1a934-d917-417b-aea3-d432d2cff990","Type":"ContainerStarted","Data":"6965aaf93fd6b13fa4c3cda4b0e13b975c5aa6bae07227a4b21d01c4ddc628aa"} Nov 03 01:27:12.157730 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:12.157707 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-jltq6" event={"ID":"1748acc6-ac8d-4a30-ba58-70ade1bcbd0f","Type":"ContainerStarted","Data":"2f046892450de428d6cbbfdaebb6ff5b88103ba4ee4748f4b2da5102ec3c592b"} Nov 03 01:27:14.040840 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:27:14.040791 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 01:27:15.175511 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:15.175409 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-jltq6" event={"ID":"1748acc6-ac8d-4a30-ba58-70ade1bcbd0f","Type":"ContainerStarted","Data":"5df2b7978496690b78ef55df0bb656324505478d5c9f332d3ed784d87d336e15"} Nov 03 01:27:15.175969 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:15.175514 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-jltq6" Nov 03 01:27:15.177081 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:15.177056 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-ghxht" event={"ID":"c4e1a934-d917-417b-aea3-d432d2cff990","Type":"ContainerStarted","Data":"e9c6e7e5b1c0fde561e930dac94e21c3d03ffda78868acd88f239c428ffa0e47"} Nov 03 01:27:15.177212 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:15.177184 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-ghxht" Nov 03 01:27:15.295243 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:15.295183 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-jltq6" podStartSLOduration=2.310084315 podStartE2EDuration="5.295169338s" totalImagesPullingTime="2.985085023s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:27:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:27:15.294350025 +0000 UTC m=+517.323832320" watchObservedRunningTime="2026-11-03 01:27:15.295169338 +0000 UTC m=+517.324651634" Nov 03 01:27:15.414216 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:15.414161 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-55fbbbf9dc-ghxht" podStartSLOduration=2.047310224 podStartE2EDuration="5.414146422s" totalImagesPullingTime="3.366836198s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:27:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:27:15.413784716 +0000 UTC m=+517.443267011" watchObservedRunningTime="2026-11-03 01:27:15.414146422 +0000 UTC m=+517.443628718" Nov 03 01:27:16.889589 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:16.889556 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:27:17.135451 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:17.135414 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-f48966749-d28rl" Nov 03 01:27:19.145290 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:19.145256 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-5d8dddb676-tvq28" Nov 03 01:27:19.509740 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:19.509654 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-7b888454db-5q4k2"] Nov 03 01:27:19.513108 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:19.513084 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7b888454db-5q4k2" Nov 03 01:27:19.567618 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:19.567573 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7b888454db-5q4k2"] Nov 03 01:27:19.592914 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:19.592870 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xjqn\" (UniqueName: \"kubernetes.io/projected/4de9a87f-21bb-4b1b-9700-0ceef4bbda2c-kube-api-access-5xjqn\") pod \"ai-gateway-controller-7b888454db-5q4k2\" (UID: \"4de9a87f-21bb-4b1b-9700-0ceef4bbda2c\") " pod="opendatahub/ai-gateway-controller-7b888454db-5q4k2" Nov 03 01:27:19.593076 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:19.592927 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4de9a87f-21bb-4b1b-9700-0ceef4bbda2c-metrics-certs\") pod \"ai-gateway-controller-7b888454db-5q4k2\" (UID: \"4de9a87f-21bb-4b1b-9700-0ceef4bbda2c\") " pod="opendatahub/ai-gateway-controller-7b888454db-5q4k2" Nov 03 01:27:19.693926 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:19.693882 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4de9a87f-21bb-4b1b-9700-0ceef4bbda2c-metrics-certs\") pod \"ai-gateway-controller-7b888454db-5q4k2\" (UID: \"4de9a87f-21bb-4b1b-9700-0ceef4bbda2c\") " pod="opendatahub/ai-gateway-controller-7b888454db-5q4k2" Nov 03 01:27:19.694107 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:19.693970 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xjqn\" (UniqueName: \"kubernetes.io/projected/4de9a87f-21bb-4b1b-9700-0ceef4bbda2c-kube-api-access-5xjqn\") pod \"ai-gateway-controller-7b888454db-5q4k2\" (UID: \"4de9a87f-21bb-4b1b-9700-0ceef4bbda2c\") " pod="opendatahub/ai-gateway-controller-7b888454db-5q4k2" Nov 03 01:27:19.696486 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:19.696449 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4de9a87f-21bb-4b1b-9700-0ceef4bbda2c-metrics-certs\") pod \"ai-gateway-controller-7b888454db-5q4k2\" (UID: \"4de9a87f-21bb-4b1b-9700-0ceef4bbda2c\") " pod="opendatahub/ai-gateway-controller-7b888454db-5q4k2" Nov 03 01:27:19.757659 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:19.757618 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xjqn\" (UniqueName: \"kubernetes.io/projected/4de9a87f-21bb-4b1b-9700-0ceef4bbda2c-kube-api-access-5xjqn\") pod \"ai-gateway-controller-7b888454db-5q4k2\" (UID: \"4de9a87f-21bb-4b1b-9700-0ceef4bbda2c\") " pod="opendatahub/ai-gateway-controller-7b888454db-5q4k2" Nov 03 01:27:19.826790 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:19.826738 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7b888454db-5q4k2" Nov 03 01:27:20.195810 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:27:20.195773 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4de9a87f_21bb_4b1b_9700_0ceef4bbda2c.slice/crio-54d383378e115ceeb3f1fb727ac9351f1a3aee6da540930c5530dd431d43a8ac WatchSource:0}: Error finding container 54d383378e115ceeb3f1fb727ac9351f1a3aee6da540930c5530dd431d43a8ac: Status 404 returned error can't find the container with id 54d383378e115ceeb3f1fb727ac9351f1a3aee6da540930c5530dd431d43a8ac Nov 03 01:27:20.197730 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:20.197701 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7b888454db-5q4k2"] Nov 03 01:27:21.204375 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:21.204336 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7b888454db-5q4k2" event={"ID":"4de9a87f-21bb-4b1b-9700-0ceef4bbda2c","Type":"ContainerStarted","Data":"32a175d5cb2ad19af8c0986a2e318edac20fdb8d18d75933356b7422ef37e526"} Nov 03 01:27:21.204798 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:21.204413 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7b888454db-5q4k2" event={"ID":"4de9a87f-21bb-4b1b-9700-0ceef4bbda2c","Type":"ContainerStarted","Data":"54d383378e115ceeb3f1fb727ac9351f1a3aee6da540930c5530dd431d43a8ac"} Nov 03 01:27:21.204798 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:21.204498 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-7b888454db-5q4k2" Nov 03 01:27:21.325302 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:21.325205 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-7b888454db-5q4k2" podStartSLOduration=1.442619515 podStartE2EDuration="2.32518506s" totalImagesPullingTime="882.565545ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:27:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:27:21.324294114 +0000 UTC m=+523.353776409" watchObservedRunningTime="2026-11-03 01:27:21.32518506 +0000 UTC m=+523.354667355" Nov 03 01:27:26.183971 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:26.183937 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-jltq6" Nov 03 01:27:26.184446 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:26.184010 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-ghxht" Nov 03 01:27:32.211850 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:32.211806 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-7b888454db-5q4k2" Nov 03 01:27:32.383942 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:32.383903 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-tvq28"] Nov 03 01:27:32.384203 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:32.384162 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-5d8dddb676-tvq28" podUID="564648fe-209b-42e7-8d01-f922683d0a3a" containerName="manager" containerID="cri-o://914e537e8463cca2312a48a5728d260c5e725521600e236cd2ab52e27f313ea0" gracePeriod=30 Nov 03 01:27:32.629993 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:32.629965 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d8dddb676-tvq28" Nov 03 01:27:32.704437 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:32.704393 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/564648fe-209b-42e7-8d01-f922683d0a3a-kube-api-access-2bbkj\" (UniqueName: \"kubernetes.io/projected/564648fe-209b-42e7-8d01-f922683d0a3a-kube-api-access-2bbkj\") pod \"564648fe-209b-42e7-8d01-f922683d0a3a\" (UID: \"564648fe-209b-42e7-8d01-f922683d0a3a\") " Nov 03 01:27:32.704637 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:32.704484 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/564648fe-209b-42e7-8d01-f922683d0a3a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/564648fe-209b-42e7-8d01-f922683d0a3a-metrics-certs\") pod \"564648fe-209b-42e7-8d01-f922683d0a3a\" (UID: \"564648fe-209b-42e7-8d01-f922683d0a3a\") " Nov 03 01:27:32.706494 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:32.706464 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/564648fe-209b-42e7-8d01-f922683d0a3a-metrics-certs" pod "564648fe-209b-42e7-8d01-f922683d0a3a" (UID: "564648fe-209b-42e7-8d01-f922683d0a3a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:27:32.706612 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:32.706510 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/564648fe-209b-42e7-8d01-f922683d0a3a-kube-api-access-2bbkj" pod "564648fe-209b-42e7-8d01-f922683d0a3a" (UID: "564648fe-209b-42e7-8d01-f922683d0a3a"). InnerVolumeSpecName "kube-api-access-2bbkj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:27:32.805206 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:32.805174 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bbkj\" (UniqueName: \"kubernetes.io/projected/564648fe-209b-42e7-8d01-f922683d0a3a-kube-api-access-2bbkj\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:27:32.805206 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:32.805208 5437 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/564648fe-209b-42e7-8d01-f922683d0a3a-metrics-certs\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:27:33.257885 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:33.257789 5437 generic.go:356] "Generic (PLEG): container finished" podID="564648fe-209b-42e7-8d01-f922683d0a3a" containerID="914e537e8463cca2312a48a5728d260c5e725521600e236cd2ab52e27f313ea0" exitCode=0 Nov 03 01:27:33.257885 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:33.257836 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d8dddb676-tvq28" Nov 03 01:27:33.257885 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:33.257869 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d8dddb676-tvq28" event={"ID":"564648fe-209b-42e7-8d01-f922683d0a3a","Type":"ContainerDied","Data":"914e537e8463cca2312a48a5728d260c5e725521600e236cd2ab52e27f313ea0"} Nov 03 01:27:33.258470 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:33.257926 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d8dddb676-tvq28" event={"ID":"564648fe-209b-42e7-8d01-f922683d0a3a","Type":"ContainerDied","Data":"844248599bca674ffa68234088b12500d21d3ced3fb2cd66870705d20bcd6373"} Nov 03 01:27:33.258470 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:33.257948 5437 scope.go:122] "RemoveContainer" containerID="914e537e8463cca2312a48a5728d260c5e725521600e236cd2ab52e27f313ea0" Nov 03 01:27:33.267223 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:33.267198 5437 scope.go:122] "RemoveContainer" containerID="914e537e8463cca2312a48a5728d260c5e725521600e236cd2ab52e27f313ea0" Nov 03 01:27:33.267586 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:27:33.267566 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"914e537e8463cca2312a48a5728d260c5e725521600e236cd2ab52e27f313ea0\": container with ID starting with 914e537e8463cca2312a48a5728d260c5e725521600e236cd2ab52e27f313ea0 not found: ID does not exist" containerID="914e537e8463cca2312a48a5728d260c5e725521600e236cd2ab52e27f313ea0" Nov 03 01:27:33.267654 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:33.267596 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"914e537e8463cca2312a48a5728d260c5e725521600e236cd2ab52e27f313ea0"} err="failed to get container status \"914e537e8463cca2312a48a5728d260c5e725521600e236cd2ab52e27f313ea0\": rpc error: code = NotFound desc = could not find container \"914e537e8463cca2312a48a5728d260c5e725521600e236cd2ab52e27f313ea0\": container with ID starting with 914e537e8463cca2312a48a5728d260c5e725521600e236cd2ab52e27f313ea0 not found: ID does not exist" Nov 03 01:27:33.408207 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:33.408176 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:27:33.426046 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:33.426007 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-tvq28"] Nov 03 01:27:33.477589 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:33.477551 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-tvq28"] Nov 03 01:27:34.088281 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:34.088240 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="564648fe-209b-42e7-8d01-f922683d0a3a" path="/var/lib/kubelet/pods/564648fe-209b-42e7-8d01-f922683d0a3a/volumes" Nov 03 01:27:35.861521 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:35.861484 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:27:36.089654 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:36.089616 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:27:37.292076 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:37.292038 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:27:38.622603 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:38.622569 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:27:42.814542 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:42.814507 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5xgkr"] Nov 03 01:27:42.815043 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:42.814913 5437 cpu_manager.go:414] "containerMap: removing container" podUID="564648fe-209b-42e7-8d01-f922683d0a3a" containerName="manager" Nov 03 01:27:42.815043 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:42.814926 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="564648fe-209b-42e7-8d01-f922683d0a3a" containerName="manager" Nov 03 01:27:42.815043 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:42.814994 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="564648fe-209b-42e7-8d01-f922683d0a3a" containerName="manager" Nov 03 01:27:42.821165 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:42.821142 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5xgkr" Nov 03 01:27:42.873170 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:42.873134 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5xgkr"] Nov 03 01:27:42.891492 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:42.891461 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e02c0d62-8c28-4b9c-9280-0843e225fe71-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5xgkr\" (UID: \"e02c0d62-8c28-4b9c-9280-0843e225fe71\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5xgkr" Nov 03 01:27:42.891644 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:42.891505 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e02c0d62-8c28-4b9c-9280-0843e225fe71-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5xgkr\" (UID: \"e02c0d62-8c28-4b9c-9280-0843e225fe71\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5xgkr" Nov 03 01:27:42.891742 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:42.891642 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e02c0d62-8c28-4b9c-9280-0843e225fe71-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5xgkr\" (UID: \"e02c0d62-8c28-4b9c-9280-0843e225fe71\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5xgkr" Nov 03 01:27:42.891742 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:42.891706 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e02c0d62-8c28-4b9c-9280-0843e225fe71-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5xgkr\" (UID: \"e02c0d62-8c28-4b9c-9280-0843e225fe71\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5xgkr" Nov 03 01:27:42.891845 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:42.891739 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e02c0d62-8c28-4b9c-9280-0843e225fe71-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5xgkr\" (UID: \"e02c0d62-8c28-4b9c-9280-0843e225fe71\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5xgkr" Nov 03 01:27:42.891845 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:42.891766 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcqvr\" (UniqueName: \"kubernetes.io/projected/e02c0d62-8c28-4b9c-9280-0843e225fe71-kube-api-access-lcqvr\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5xgkr\" (UID: \"e02c0d62-8c28-4b9c-9280-0843e225fe71\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5xgkr" Nov 03 01:27:42.923702 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:42.923665 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Nov 03 01:27:42.923702 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:42.923673 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-j8f5n\"" Nov 03 01:27:42.923901 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:42.923672 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Nov 03 01:27:42.923901 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:42.923714 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Nov 03 01:27:42.963449 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:42.963415 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:27:42.992709 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:42.992673 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e02c0d62-8c28-4b9c-9280-0843e225fe71-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5xgkr\" (UID: \"e02c0d62-8c28-4b9c-9280-0843e225fe71\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5xgkr" Nov 03 01:27:42.992709 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:42.992713 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e02c0d62-8c28-4b9c-9280-0843e225fe71-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5xgkr\" (UID: \"e02c0d62-8c28-4b9c-9280-0843e225fe71\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5xgkr" Nov 03 01:27:42.992951 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:42.992740 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e02c0d62-8c28-4b9c-9280-0843e225fe71-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5xgkr\" (UID: \"e02c0d62-8c28-4b9c-9280-0843e225fe71\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5xgkr" Nov 03 01:27:42.992951 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:42.992768 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcqvr\" (UniqueName: \"kubernetes.io/projected/e02c0d62-8c28-4b9c-9280-0843e225fe71-kube-api-access-lcqvr\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5xgkr\" (UID: \"e02c0d62-8c28-4b9c-9280-0843e225fe71\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5xgkr" Nov 03 01:27:42.992951 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:42.992831 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e02c0d62-8c28-4b9c-9280-0843e225fe71-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5xgkr\" (UID: \"e02c0d62-8c28-4b9c-9280-0843e225fe71\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5xgkr" Nov 03 01:27:42.992951 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:42.992867 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e02c0d62-8c28-4b9c-9280-0843e225fe71-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5xgkr\" (UID: \"e02c0d62-8c28-4b9c-9280-0843e225fe71\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5xgkr" Nov 03 01:27:42.993180 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:42.993156 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e02c0d62-8c28-4b9c-9280-0843e225fe71-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5xgkr\" (UID: \"e02c0d62-8c28-4b9c-9280-0843e225fe71\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5xgkr" Nov 03 01:27:42.993236 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:42.993200 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e02c0d62-8c28-4b9c-9280-0843e225fe71-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5xgkr\" (UID: \"e02c0d62-8c28-4b9c-9280-0843e225fe71\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5xgkr" Nov 03 01:27:42.993292 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:42.993256 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e02c0d62-8c28-4b9c-9280-0843e225fe71-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5xgkr\" (UID: \"e02c0d62-8c28-4b9c-9280-0843e225fe71\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5xgkr" Nov 03 01:27:42.995070 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:42.995048 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e02c0d62-8c28-4b9c-9280-0843e225fe71-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5xgkr\" (UID: \"e02c0d62-8c28-4b9c-9280-0843e225fe71\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5xgkr" Nov 03 01:27:42.995310 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:42.995291 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e02c0d62-8c28-4b9c-9280-0843e225fe71-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5xgkr\" (UID: \"e02c0d62-8c28-4b9c-9280-0843e225fe71\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5xgkr" Nov 03 01:27:43.056105 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:43.056064 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcqvr\" (UniqueName: \"kubernetes.io/projected/e02c0d62-8c28-4b9c-9280-0843e225fe71-kube-api-access-lcqvr\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5xgkr\" (UID: \"e02c0d62-8c28-4b9c-9280-0843e225fe71\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5xgkr" Nov 03 01:27:43.135218 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:43.135119 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5xgkr" Nov 03 01:27:43.512926 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:27:43.512894 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode02c0d62_8c28_4b9c_9280_0843e225fe71.slice/crio-f4e2053494da61bcf88f9c3455c56245c20df82815f3b56354229efd0f428d5e WatchSource:0}: Error finding container f4e2053494da61bcf88f9c3455c56245c20df82815f3b56354229efd0f428d5e: Status 404 returned error can't find the container with id f4e2053494da61bcf88f9c3455c56245c20df82815f3b56354229efd0f428d5e Nov 03 01:27:43.514258 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:43.514239 5437 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 01:27:43.514929 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:43.514901 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5xgkr"] Nov 03 01:27:44.035634 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:27:44.035597 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 01:27:44.310172 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:44.310138 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5xgkr" event={"ID":"e02c0d62-8c28-4b9c-9280-0843e225fe71","Type":"ContainerStarted","Data":"f4e2053494da61bcf88f9c3455c56245c20df82815f3b56354229efd0f428d5e"} Nov 03 01:27:45.367065 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:45.367029 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:27:46.320495 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:46.320457 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5xgkr" event={"ID":"e02c0d62-8c28-4b9c-9280-0843e225fe71","Type":"ContainerStarted","Data":"2271cc49066ede8c06d65ec154f82af24814142738279dc3c7b1b3e731e78b27"} Nov 03 01:27:46.442076 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:46.442020 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5xgkr" podStartSLOduration=2.055359558 podStartE2EDuration="4.442004142s" totalImagesPullingTime="2.386644584s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:27:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:27:46.441493901 +0000 UTC m=+548.470976196" watchObservedRunningTime="2026-11-03 01:27:46.442004142 +0000 UTC m=+548.471486436" Nov 03 01:27:47.135959 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:47.135874 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5xgkr" Nov 03 01:27:49.268736 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:49.268696 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-qknt5"] Nov 03 01:27:49.272149 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:49.272127 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qknt5" Nov 03 01:27:49.326933 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:49.326891 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Nov 03 01:27:49.352835 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:49.352801 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b3bf52c-b366-481a-802f-4c2cc0384280-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-qknt5\" (UID: \"7b3bf52c-b366-481a-802f-4c2cc0384280\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qknt5" Nov 03 01:27:49.353004 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:49.352848 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b3bf52c-b366-481a-802f-4c2cc0384280-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-qknt5\" (UID: \"7b3bf52c-b366-481a-802f-4c2cc0384280\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qknt5" Nov 03 01:27:49.353004 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:49.352920 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b3bf52c-b366-481a-802f-4c2cc0384280-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-qknt5\" (UID: \"7b3bf52c-b366-481a-802f-4c2cc0384280\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qknt5" Nov 03 01:27:49.353004 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:49.352976 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b3bf52c-b366-481a-802f-4c2cc0384280-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-qknt5\" (UID: \"7b3bf52c-b366-481a-802f-4c2cc0384280\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qknt5" Nov 03 01:27:49.353143 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:49.353010 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b3bf52c-b366-481a-802f-4c2cc0384280-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-qknt5\" (UID: \"7b3bf52c-b366-481a-802f-4c2cc0384280\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qknt5" Nov 03 01:27:49.353143 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:49.353028 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26mrw\" (UniqueName: \"kubernetes.io/projected/7b3bf52c-b366-481a-802f-4c2cc0384280-kube-api-access-26mrw\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-qknt5\" (UID: \"7b3bf52c-b366-481a-802f-4c2cc0384280\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qknt5" Nov 03 01:27:49.379432 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:49.379400 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-qknt5"] Nov 03 01:27:49.433649 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:49.433617 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:27:49.453965 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:49.453921 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b3bf52c-b366-481a-802f-4c2cc0384280-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-qknt5\" (UID: \"7b3bf52c-b366-481a-802f-4c2cc0384280\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qknt5" Nov 03 01:27:49.453965 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:49.453967 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b3bf52c-b366-481a-802f-4c2cc0384280-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-qknt5\" (UID: \"7b3bf52c-b366-481a-802f-4c2cc0384280\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qknt5" Nov 03 01:27:49.454182 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:49.454085 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b3bf52c-b366-481a-802f-4c2cc0384280-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-qknt5\" (UID: \"7b3bf52c-b366-481a-802f-4c2cc0384280\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qknt5" Nov 03 01:27:49.454182 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:49.454139 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b3bf52c-b366-481a-802f-4c2cc0384280-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-qknt5\" (UID: \"7b3bf52c-b366-481a-802f-4c2cc0384280\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qknt5" Nov 03 01:27:49.454281 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:49.454256 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b3bf52c-b366-481a-802f-4c2cc0384280-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-qknt5\" (UID: \"7b3bf52c-b366-481a-802f-4c2cc0384280\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qknt5" Nov 03 01:27:49.454339 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:49.454297 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26mrw\" (UniqueName: \"kubernetes.io/projected/7b3bf52c-b366-481a-802f-4c2cc0384280-kube-api-access-26mrw\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-qknt5\" (UID: \"7b3bf52c-b366-481a-802f-4c2cc0384280\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qknt5" Nov 03 01:27:49.454487 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:49.454460 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b3bf52c-b366-481a-802f-4c2cc0384280-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-qknt5\" (UID: \"7b3bf52c-b366-481a-802f-4c2cc0384280\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qknt5" Nov 03 01:27:49.454487 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:49.454482 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b3bf52c-b366-481a-802f-4c2cc0384280-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-qknt5\" (UID: \"7b3bf52c-b366-481a-802f-4c2cc0384280\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qknt5" Nov 03 01:27:49.454691 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:49.454672 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b3bf52c-b366-481a-802f-4c2cc0384280-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-qknt5\" (UID: \"7b3bf52c-b366-481a-802f-4c2cc0384280\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qknt5" Nov 03 01:27:49.456279 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:49.456261 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b3bf52c-b366-481a-802f-4c2cc0384280-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-qknt5\" (UID: \"7b3bf52c-b366-481a-802f-4c2cc0384280\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qknt5" Nov 03 01:27:49.456566 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:49.456550 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b3bf52c-b366-481a-802f-4c2cc0384280-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-qknt5\" (UID: \"7b3bf52c-b366-481a-802f-4c2cc0384280\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qknt5" Nov 03 01:27:49.517277 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:49.517241 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26mrw\" (UniqueName: \"kubernetes.io/projected/7b3bf52c-b366-481a-802f-4c2cc0384280-kube-api-access-26mrw\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-qknt5\" (UID: \"7b3bf52c-b366-481a-802f-4c2cc0384280\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qknt5" Nov 03 01:27:49.584486 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:49.584445 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qknt5" Nov 03 01:27:49.951720 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:27:49.951690 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b3bf52c_b366_481a_802f_4c2cc0384280.slice/crio-2d2489bade1a3e97d3b34f073841237e9ca97583ed430ac1fd12ca02fd6b0d63 WatchSource:0}: Error finding container 2d2489bade1a3e97d3b34f073841237e9ca97583ed430ac1fd12ca02fd6b0d63: Status 404 returned error can't find the container with id 2d2489bade1a3e97d3b34f073841237e9ca97583ed430ac1fd12ca02fd6b0d63 Nov 03 01:27:49.953520 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:49.953492 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-qknt5"] Nov 03 01:27:50.338648 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:50.338603 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qknt5" event={"ID":"7b3bf52c-b366-481a-802f-4c2cc0384280","Type":"ContainerStarted","Data":"2d2489bade1a3e97d3b34f073841237e9ca97583ed430ac1fd12ca02fd6b0d63"} Nov 03 01:27:51.344650 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:51.344613 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qknt5" event={"ID":"7b3bf52c-b366-481a-802f-4c2cc0384280","Type":"ContainerStarted","Data":"7cbb395e784ed760fa4f0e7a324f4f57d095e6788c30631670a3c8a707896148"} Nov 03 01:27:51.465881 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:51.465813 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qknt5" podStartSLOduration=1.8271109540000001 podStartE2EDuration="2.465797035s" totalImagesPullingTime="638.686081ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:27:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:27:51.46488885 +0000 UTC m=+553.494371145" watchObservedRunningTime="2026-11-03 01:27:51.465797035 +0000 UTC m=+553.495279384" Nov 03 01:27:51.585451 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:51.585410 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qknt5" Nov 03 01:27:52.688583 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:52.688543 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-rpmbj"] Nov 03 01:27:52.691935 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:52.691912 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rpmbj" Nov 03 01:27:52.776402 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:52.776346 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-rpmbj"] Nov 03 01:27:52.785606 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:52.785575 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a552ecb-43a3-4154-b818-24e78b95b4f1-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-rpmbj\" (UID: \"9a552ecb-43a3-4154-b818-24e78b95b4f1\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rpmbj" Nov 03 01:27:52.785724 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:52.785626 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a552ecb-43a3-4154-b818-24e78b95b4f1-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-rpmbj\" (UID: \"9a552ecb-43a3-4154-b818-24e78b95b4f1\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rpmbj" Nov 03 01:27:52.785787 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:52.785730 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a552ecb-43a3-4154-b818-24e78b95b4f1-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-rpmbj\" (UID: \"9a552ecb-43a3-4154-b818-24e78b95b4f1\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rpmbj" Nov 03 01:27:52.785848 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:52.785829 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czhpg\" (UniqueName: \"kubernetes.io/projected/9a552ecb-43a3-4154-b818-24e78b95b4f1-kube-api-access-czhpg\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-rpmbj\" (UID: \"9a552ecb-43a3-4154-b818-24e78b95b4f1\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rpmbj" Nov 03 01:27:52.785903 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:52.785862 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a552ecb-43a3-4154-b818-24e78b95b4f1-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-rpmbj\" (UID: \"9a552ecb-43a3-4154-b818-24e78b95b4f1\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rpmbj" Nov 03 01:27:52.785956 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:52.785926 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a552ecb-43a3-4154-b818-24e78b95b4f1-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-rpmbj\" (UID: \"9a552ecb-43a3-4154-b818-24e78b95b4f1\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rpmbj" Nov 03 01:27:52.798319 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:52.798295 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Nov 03 01:27:52.848659 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:52.848622 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:27:52.886688 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:52.886648 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a552ecb-43a3-4154-b818-24e78b95b4f1-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-rpmbj\" (UID: \"9a552ecb-43a3-4154-b818-24e78b95b4f1\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rpmbj" Nov 03 01:27:52.886878 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:52.886710 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czhpg\" (UniqueName: \"kubernetes.io/projected/9a552ecb-43a3-4154-b818-24e78b95b4f1-kube-api-access-czhpg\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-rpmbj\" (UID: \"9a552ecb-43a3-4154-b818-24e78b95b4f1\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rpmbj" Nov 03 01:27:52.886878 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:52.886737 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a552ecb-43a3-4154-b818-24e78b95b4f1-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-rpmbj\" (UID: \"9a552ecb-43a3-4154-b818-24e78b95b4f1\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rpmbj" Nov 03 01:27:52.886878 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:52.886775 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a552ecb-43a3-4154-b818-24e78b95b4f1-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-rpmbj\" (UID: \"9a552ecb-43a3-4154-b818-24e78b95b4f1\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rpmbj" Nov 03 01:27:52.887020 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:52.886930 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a552ecb-43a3-4154-b818-24e78b95b4f1-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-rpmbj\" (UID: \"9a552ecb-43a3-4154-b818-24e78b95b4f1\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rpmbj" Nov 03 01:27:52.887020 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:52.887000 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a552ecb-43a3-4154-b818-24e78b95b4f1-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-rpmbj\" (UID: \"9a552ecb-43a3-4154-b818-24e78b95b4f1\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rpmbj" Nov 03 01:27:52.887146 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:52.887127 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a552ecb-43a3-4154-b818-24e78b95b4f1-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-rpmbj\" (UID: \"9a552ecb-43a3-4154-b818-24e78b95b4f1\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rpmbj" Nov 03 01:27:52.887203 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:52.887147 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a552ecb-43a3-4154-b818-24e78b95b4f1-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-rpmbj\" (UID: \"9a552ecb-43a3-4154-b818-24e78b95b4f1\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rpmbj" Nov 03 01:27:52.887412 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:52.887392 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a552ecb-43a3-4154-b818-24e78b95b4f1-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-rpmbj\" (UID: \"9a552ecb-43a3-4154-b818-24e78b95b4f1\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rpmbj" Nov 03 01:27:52.889219 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:52.889193 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a552ecb-43a3-4154-b818-24e78b95b4f1-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-rpmbj\" (UID: \"9a552ecb-43a3-4154-b818-24e78b95b4f1\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rpmbj" Nov 03 01:27:52.889411 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:52.889392 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a552ecb-43a3-4154-b818-24e78b95b4f1-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-rpmbj\" (UID: \"9a552ecb-43a3-4154-b818-24e78b95b4f1\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rpmbj" Nov 03 01:27:52.950243 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:52.950160 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czhpg\" (UniqueName: \"kubernetes.io/projected/9a552ecb-43a3-4154-b818-24e78b95b4f1-kube-api-access-czhpg\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-rpmbj\" (UID: \"9a552ecb-43a3-4154-b818-24e78b95b4f1\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rpmbj" Nov 03 01:27:53.003877 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:53.003835 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rpmbj" Nov 03 01:27:53.135533 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:53.135489 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5xgkr" Nov 03 01:27:53.148040 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:53.148013 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5xgkr" Nov 03 01:27:53.365651 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:53.365620 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5xgkr" Nov 03 01:27:53.373966 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:27:53.373926 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a552ecb_43a3_4154_b818_24e78b95b4f1.slice/crio-a4d35af12bc3ef6992e13efc7b83c1aa4ae82926c609eb043f6cc6a855323845 WatchSource:0}: Error finding container a4d35af12bc3ef6992e13efc7b83c1aa4ae82926c609eb043f6cc6a855323845: Status 404 returned error can't find the container with id a4d35af12bc3ef6992e13efc7b83c1aa4ae82926c609eb043f6cc6a855323845 Nov 03 01:27:53.375794 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:53.375768 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-rpmbj"] Nov 03 01:27:54.358645 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:54.358606 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rpmbj" event={"ID":"9a552ecb-43a3-4154-b818-24e78b95b4f1","Type":"ContainerStarted","Data":"929a42b1a5591928f71d6349c40a1cf434863f02b7e4904ff64a62b92d658ff6"} Nov 03 01:27:54.359134 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:54.358664 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rpmbj" event={"ID":"9a552ecb-43a3-4154-b818-24e78b95b4f1","Type":"ContainerStarted","Data":"a4d35af12bc3ef6992e13efc7b83c1aa4ae82926c609eb043f6cc6a855323845"} Nov 03 01:27:54.480600 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:54.480544 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rpmbj" podStartSLOduration=1.8658478600000001 podStartE2EDuration="2.480529369s" totalImagesPullingTime="614.681509ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:27:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:27:54.479201243 +0000 UTC m=+556.508683538" watchObservedRunningTime="2026-11-03 01:27:54.480529369 +0000 UTC m=+556.510011663" Nov 03 01:27:55.004620 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:55.004577 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rpmbj" Nov 03 01:27:57.578179 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:57.578144 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-jthts"] Nov 03 01:27:57.581868 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:57.581849 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jthts" Nov 03 01:27:57.637667 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:57.637632 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-jthts"] Nov 03 01:27:57.687708 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:57.687667 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Nov 03 01:27:57.729426 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:57.729384 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:27:57.731462 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:57.731435 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-454n6\" (UniqueName: \"kubernetes.io/projected/7a292fcf-e87a-431c-81fe-33a292fc2d0a-kube-api-access-454n6\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-jthts\" (UID: \"7a292fcf-e87a-431c-81fe-33a292fc2d0a\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jthts" Nov 03 01:27:57.731607 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:57.731478 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a292fcf-e87a-431c-81fe-33a292fc2d0a-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-jthts\" (UID: \"7a292fcf-e87a-431c-81fe-33a292fc2d0a\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jthts" Nov 03 01:27:57.731607 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:57.731552 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a292fcf-e87a-431c-81fe-33a292fc2d0a-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-jthts\" (UID: \"7a292fcf-e87a-431c-81fe-33a292fc2d0a\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jthts" Nov 03 01:27:57.731607 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:57.731589 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a292fcf-e87a-431c-81fe-33a292fc2d0a-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-jthts\" (UID: \"7a292fcf-e87a-431c-81fe-33a292fc2d0a\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jthts" Nov 03 01:27:57.731724 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:57.731705 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a292fcf-e87a-431c-81fe-33a292fc2d0a-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-jthts\" (UID: \"7a292fcf-e87a-431c-81fe-33a292fc2d0a\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jthts" Nov 03 01:27:57.731773 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:57.731732 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a292fcf-e87a-431c-81fe-33a292fc2d0a-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-jthts\" (UID: \"7a292fcf-e87a-431c-81fe-33a292fc2d0a\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jthts" Nov 03 01:27:57.832935 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:57.832812 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-454n6\" (UniqueName: \"kubernetes.io/projected/7a292fcf-e87a-431c-81fe-33a292fc2d0a-kube-api-access-454n6\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-jthts\" (UID: \"7a292fcf-e87a-431c-81fe-33a292fc2d0a\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jthts" Nov 03 01:27:57.832935 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:57.832870 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a292fcf-e87a-431c-81fe-33a292fc2d0a-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-jthts\" (UID: \"7a292fcf-e87a-431c-81fe-33a292fc2d0a\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jthts" Nov 03 01:27:57.832935 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:57.832904 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a292fcf-e87a-431c-81fe-33a292fc2d0a-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-jthts\" (UID: \"7a292fcf-e87a-431c-81fe-33a292fc2d0a\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jthts" Nov 03 01:27:57.832935 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:57.832938 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a292fcf-e87a-431c-81fe-33a292fc2d0a-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-jthts\" (UID: \"7a292fcf-e87a-431c-81fe-33a292fc2d0a\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jthts" Nov 03 01:27:57.833508 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:57.833010 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a292fcf-e87a-431c-81fe-33a292fc2d0a-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-jthts\" (UID: \"7a292fcf-e87a-431c-81fe-33a292fc2d0a\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jthts" Nov 03 01:27:57.833508 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:57.833034 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a292fcf-e87a-431c-81fe-33a292fc2d0a-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-jthts\" (UID: \"7a292fcf-e87a-431c-81fe-33a292fc2d0a\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jthts" Nov 03 01:27:57.833508 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:57.833414 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a292fcf-e87a-431c-81fe-33a292fc2d0a-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-jthts\" (UID: \"7a292fcf-e87a-431c-81fe-33a292fc2d0a\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jthts" Nov 03 01:27:57.833508 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:57.833440 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a292fcf-e87a-431c-81fe-33a292fc2d0a-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-jthts\" (UID: \"7a292fcf-e87a-431c-81fe-33a292fc2d0a\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jthts" Nov 03 01:27:57.833666 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:57.833509 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a292fcf-e87a-431c-81fe-33a292fc2d0a-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-jthts\" (UID: \"7a292fcf-e87a-431c-81fe-33a292fc2d0a\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jthts" Nov 03 01:27:57.835155 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:57.835131 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a292fcf-e87a-431c-81fe-33a292fc2d0a-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-jthts\" (UID: \"7a292fcf-e87a-431c-81fe-33a292fc2d0a\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jthts" Nov 03 01:27:57.835457 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:57.835439 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a292fcf-e87a-431c-81fe-33a292fc2d0a-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-jthts\" (UID: \"7a292fcf-e87a-431c-81fe-33a292fc2d0a\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jthts" Nov 03 01:27:57.896185 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:57.896154 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-454n6\" (UniqueName: \"kubernetes.io/projected/7a292fcf-e87a-431c-81fe-33a292fc2d0a-kube-api-access-454n6\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-jthts\" (UID: \"7a292fcf-e87a-431c-81fe-33a292fc2d0a\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jthts" Nov 03 01:27:58.194413 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:58.194276 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jthts" Nov 03 01:27:58.565419 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:27:58.565387 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a292fcf_e87a_431c_81fe_33a292fc2d0a.slice/crio-407ccbc4e22fb3a3953e2ef6c46d0e4c69e3a29241c89e24226ce5d676c9c1e0 WatchSource:0}: Error finding container 407ccbc4e22fb3a3953e2ef6c46d0e4c69e3a29241c89e24226ce5d676c9c1e0: Status 404 returned error can't find the container with id 407ccbc4e22fb3a3953e2ef6c46d0e4c69e3a29241c89e24226ce5d676c9c1e0 Nov 03 01:27:58.567461 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:58.567430 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-jthts"] Nov 03 01:27:59.381959 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:59.381922 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jthts" event={"ID":"7a292fcf-e87a-431c-81fe-33a292fc2d0a","Type":"ContainerStarted","Data":"e3648e7b1c81bb52758304077f58554d81b4675958a05367b4120716eb5086a5"} Nov 03 01:27:59.382422 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:59.381973 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jthts" event={"ID":"7a292fcf-e87a-431c-81fe-33a292fc2d0a","Type":"ContainerStarted","Data":"407ccbc4e22fb3a3953e2ef6c46d0e4c69e3a29241c89e24226ce5d676c9c1e0"} Nov 03 01:27:59.507088 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:59.506967 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jthts" podStartSLOduration=1.8856089919999999 podStartE2EDuration="2.506951611s" totalImagesPullingTime="621.342619ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:27:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:27:59.506313407 +0000 UTC m=+561.535795703" watchObservedRunningTime="2026-11-03 01:27:59.506951611 +0000 UTC m=+561.536433905" Nov 03 01:27:59.584658 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:59.584610 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qknt5" Nov 03 01:27:59.597526 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:27:59.597490 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qknt5" Nov 03 01:28:00.194549 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:00.194508 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jthts" Nov 03 01:28:00.397875 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:00.397842 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qknt5" Nov 03 01:28:02.089832 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:02.089799 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qrnjn"] Nov 03 01:28:02.093299 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:02.093274 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qrnjn" Nov 03 01:28:02.145122 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:02.145092 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qrnjn"] Nov 03 01:28:02.170143 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:02.170090 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/835a3491-fa81-4354-924f-4d9ef651e58e-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qrnjn\" (UID: \"835a3491-fa81-4354-924f-4d9ef651e58e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qrnjn" Nov 03 01:28:02.170332 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:02.170254 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/835a3491-fa81-4354-924f-4d9ef651e58e-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qrnjn\" (UID: \"835a3491-fa81-4354-924f-4d9ef651e58e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qrnjn" Nov 03 01:28:02.170427 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:02.170349 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/835a3491-fa81-4354-924f-4d9ef651e58e-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qrnjn\" (UID: \"835a3491-fa81-4354-924f-4d9ef651e58e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qrnjn" Nov 03 01:28:02.170493 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:02.170453 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jh7s7\" (UniqueName: \"kubernetes.io/projected/835a3491-fa81-4354-924f-4d9ef651e58e-kube-api-access-jh7s7\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qrnjn\" (UID: \"835a3491-fa81-4354-924f-4d9ef651e58e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qrnjn" Nov 03 01:28:02.170560 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:02.170544 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/835a3491-fa81-4354-924f-4d9ef651e58e-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qrnjn\" (UID: \"835a3491-fa81-4354-924f-4d9ef651e58e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qrnjn" Nov 03 01:28:02.170623 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:02.170596 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/835a3491-fa81-4354-924f-4d9ef651e58e-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qrnjn\" (UID: \"835a3491-fa81-4354-924f-4d9ef651e58e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qrnjn" Nov 03 01:28:02.195512 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:02.195480 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Nov 03 01:28:02.238807 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:02.238775 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:02.271808 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:02.271770 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/835a3491-fa81-4354-924f-4d9ef651e58e-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qrnjn\" (UID: \"835a3491-fa81-4354-924f-4d9ef651e58e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qrnjn" Nov 03 01:28:02.271991 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:02.271823 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/835a3491-fa81-4354-924f-4d9ef651e58e-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qrnjn\" (UID: \"835a3491-fa81-4354-924f-4d9ef651e58e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qrnjn" Nov 03 01:28:02.271991 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:02.271848 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jh7s7\" (UniqueName: \"kubernetes.io/projected/835a3491-fa81-4354-924f-4d9ef651e58e-kube-api-access-jh7s7\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qrnjn\" (UID: \"835a3491-fa81-4354-924f-4d9ef651e58e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qrnjn" Nov 03 01:28:02.271991 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:02.271902 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/835a3491-fa81-4354-924f-4d9ef651e58e-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qrnjn\" (UID: \"835a3491-fa81-4354-924f-4d9ef651e58e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qrnjn" Nov 03 01:28:02.271991 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:02.271923 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/835a3491-fa81-4354-924f-4d9ef651e58e-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qrnjn\" (UID: \"835a3491-fa81-4354-924f-4d9ef651e58e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qrnjn" Nov 03 01:28:02.271991 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:02.271951 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/835a3491-fa81-4354-924f-4d9ef651e58e-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qrnjn\" (UID: \"835a3491-fa81-4354-924f-4d9ef651e58e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qrnjn" Nov 03 01:28:02.272313 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:02.272282 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/835a3491-fa81-4354-924f-4d9ef651e58e-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qrnjn\" (UID: \"835a3491-fa81-4354-924f-4d9ef651e58e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qrnjn" Nov 03 01:28:02.272390 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:02.272345 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/835a3491-fa81-4354-924f-4d9ef651e58e-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qrnjn\" (UID: \"835a3491-fa81-4354-924f-4d9ef651e58e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qrnjn" Nov 03 01:28:02.276410 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:02.272870 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/835a3491-fa81-4354-924f-4d9ef651e58e-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qrnjn\" (UID: \"835a3491-fa81-4354-924f-4d9ef651e58e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qrnjn" Nov 03 01:28:02.276410 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:02.274711 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/835a3491-fa81-4354-924f-4d9ef651e58e-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qrnjn\" (UID: \"835a3491-fa81-4354-924f-4d9ef651e58e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qrnjn" Nov 03 01:28:02.279501 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:02.279392 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/835a3491-fa81-4354-924f-4d9ef651e58e-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qrnjn\" (UID: \"835a3491-fa81-4354-924f-4d9ef651e58e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qrnjn" Nov 03 01:28:02.335190 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:02.335151 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jh7s7\" (UniqueName: \"kubernetes.io/projected/835a3491-fa81-4354-924f-4d9ef651e58e-kube-api-access-jh7s7\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qrnjn\" (UID: \"835a3491-fa81-4354-924f-4d9ef651e58e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qrnjn" Nov 03 01:28:02.405632 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:02.405543 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qrnjn" Nov 03 01:28:02.774239 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:28:02.774199 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod835a3491_fa81_4354_924f_4d9ef651e58e.slice/crio-1f4d33e94a1863ae0761641b7ad9ef0a1e1925a63e21c37a758660bf4dc17307 WatchSource:0}: Error finding container 1f4d33e94a1863ae0761641b7ad9ef0a1e1925a63e21c37a758660bf4dc17307: Status 404 returned error can't find the container with id 1f4d33e94a1863ae0761641b7ad9ef0a1e1925a63e21c37a758660bf4dc17307 Nov 03 01:28:02.775800 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:02.775774 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qrnjn"] Nov 03 01:28:03.004840 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:03.004798 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rpmbj" Nov 03 01:28:03.017398 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:03.017343 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rpmbj" Nov 03 01:28:03.403492 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:03.403456 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qrnjn" event={"ID":"835a3491-fa81-4354-924f-4d9ef651e58e","Type":"ContainerStarted","Data":"1f4d33e94a1863ae0761641b7ad9ef0a1e1925a63e21c37a758660bf4dc17307"} Nov 03 01:28:03.415458 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:03.415422 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rpmbj" Nov 03 01:28:04.410502 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:04.410462 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qrnjn" event={"ID":"835a3491-fa81-4354-924f-4d9ef651e58e","Type":"ContainerStarted","Data":"f30135875d981e136705cec327343c20df80523001903ed9b2d0c06a0448ea9f"} Nov 03 01:28:04.532465 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:04.532405 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qrnjn" podStartSLOduration=1.922242247 podStartE2EDuration="2.532389564s" totalImagesPullingTime="610.147317ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:28:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:28:04.531802372 +0000 UTC m=+566.561284667" watchObservedRunningTime="2026-11-03 01:28:04.532389564 +0000 UTC m=+566.561871856" Nov 03 01:28:05.143126 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:05.143069 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:05.406717 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:05.406635 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qrnjn" Nov 03 01:28:08.194575 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:08.194534 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jthts" Nov 03 01:28:08.207338 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:08.207314 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jthts" Nov 03 01:28:08.430820 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:08.430785 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-h9gfn"] Nov 03 01:28:08.434190 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:08.434171 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-h9gfn" Nov 03 01:28:08.441517 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:08.441494 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jthts" Nov 03 01:28:08.488599 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:08.488524 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Nov 03 01:28:08.489971 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:08.489943 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-h9gfn"] Nov 03 01:28:08.532884 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:08.532856 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4d4e0c0-2ae5-402d-a47a-d8e4c79d7cfa-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-h9gfn\" (UID: \"d4d4e0c0-2ae5-402d-a47a-d8e4c79d7cfa\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-h9gfn" Nov 03 01:28:08.533013 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:08.532917 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4d4e0c0-2ae5-402d-a47a-d8e4c79d7cfa-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-h9gfn\" (UID: \"d4d4e0c0-2ae5-402d-a47a-d8e4c79d7cfa\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-h9gfn" Nov 03 01:28:08.533287 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:08.533025 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4d4e0c0-2ae5-402d-a47a-d8e4c79d7cfa-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-h9gfn\" (UID: \"d4d4e0c0-2ae5-402d-a47a-d8e4c79d7cfa\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-h9gfn" Nov 03 01:28:08.533287 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:08.533059 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4d4e0c0-2ae5-402d-a47a-d8e4c79d7cfa-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-h9gfn\" (UID: \"d4d4e0c0-2ae5-402d-a47a-d8e4c79d7cfa\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-h9gfn" Nov 03 01:28:08.533287 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:08.533081 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4d4e0c0-2ae5-402d-a47a-d8e4c79d7cfa-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-h9gfn\" (UID: \"d4d4e0c0-2ae5-402d-a47a-d8e4c79d7cfa\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-h9gfn" Nov 03 01:28:08.533287 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:08.533226 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqncs\" (UniqueName: \"kubernetes.io/projected/d4d4e0c0-2ae5-402d-a47a-d8e4c79d7cfa-kube-api-access-hqncs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-h9gfn\" (UID: \"d4d4e0c0-2ae5-402d-a47a-d8e4c79d7cfa\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-h9gfn" Nov 03 01:28:08.583057 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:08.583026 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:08.634372 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:08.634331 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqncs\" (UniqueName: \"kubernetes.io/projected/d4d4e0c0-2ae5-402d-a47a-d8e4c79d7cfa-kube-api-access-hqncs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-h9gfn\" (UID: \"d4d4e0c0-2ae5-402d-a47a-d8e4c79d7cfa\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-h9gfn" Nov 03 01:28:08.634518 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:08.634449 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4d4e0c0-2ae5-402d-a47a-d8e4c79d7cfa-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-h9gfn\" (UID: \"d4d4e0c0-2ae5-402d-a47a-d8e4c79d7cfa\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-h9gfn" Nov 03 01:28:08.634518 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:08.634505 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4d4e0c0-2ae5-402d-a47a-d8e4c79d7cfa-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-h9gfn\" (UID: \"d4d4e0c0-2ae5-402d-a47a-d8e4c79d7cfa\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-h9gfn" Nov 03 01:28:08.634611 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:08.634558 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4d4e0c0-2ae5-402d-a47a-d8e4c79d7cfa-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-h9gfn\" (UID: \"d4d4e0c0-2ae5-402d-a47a-d8e4c79d7cfa\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-h9gfn" Nov 03 01:28:08.634611 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:08.634587 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4d4e0c0-2ae5-402d-a47a-d8e4c79d7cfa-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-h9gfn\" (UID: \"d4d4e0c0-2ae5-402d-a47a-d8e4c79d7cfa\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-h9gfn" Nov 03 01:28:08.634722 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:08.634610 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4d4e0c0-2ae5-402d-a47a-d8e4c79d7cfa-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-h9gfn\" (UID: \"d4d4e0c0-2ae5-402d-a47a-d8e4c79d7cfa\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-h9gfn" Nov 03 01:28:08.634999 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:08.634976 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4d4e0c0-2ae5-402d-a47a-d8e4c79d7cfa-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-h9gfn\" (UID: \"d4d4e0c0-2ae5-402d-a47a-d8e4c79d7cfa\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-h9gfn" Nov 03 01:28:08.635104 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:08.635050 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4d4e0c0-2ae5-402d-a47a-d8e4c79d7cfa-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-h9gfn\" (UID: \"d4d4e0c0-2ae5-402d-a47a-d8e4c79d7cfa\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-h9gfn" Nov 03 01:28:08.635104 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:08.635049 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4d4e0c0-2ae5-402d-a47a-d8e4c79d7cfa-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-h9gfn\" (UID: \"d4d4e0c0-2ae5-402d-a47a-d8e4c79d7cfa\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-h9gfn" Nov 03 01:28:08.636760 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:08.636739 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4d4e0c0-2ae5-402d-a47a-d8e4c79d7cfa-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-h9gfn\" (UID: \"d4d4e0c0-2ae5-402d-a47a-d8e4c79d7cfa\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-h9gfn" Nov 03 01:28:08.637307 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:08.637286 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4d4e0c0-2ae5-402d-a47a-d8e4c79d7cfa-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-h9gfn\" (UID: \"d4d4e0c0-2ae5-402d-a47a-d8e4c79d7cfa\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-h9gfn" Nov 03 01:28:08.697966 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:08.697931 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqncs\" (UniqueName: \"kubernetes.io/projected/d4d4e0c0-2ae5-402d-a47a-d8e4c79d7cfa-kube-api-access-hqncs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-h9gfn\" (UID: \"d4d4e0c0-2ae5-402d-a47a-d8e4c79d7cfa\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-h9gfn" Nov 03 01:28:08.746678 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:08.746595 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-h9gfn" Nov 03 01:28:09.111432 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:28:09.111399 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4d4e0c0_2ae5_402d_a47a_d8e4c79d7cfa.slice/crio-3f05b45878b69de1d144b0f7cc861075459bcd68e870551241c13dd92f623fdc WatchSource:0}: Error finding container 3f05b45878b69de1d144b0f7cc861075459bcd68e870551241c13dd92f623fdc: Status 404 returned error can't find the container with id 3f05b45878b69de1d144b0f7cc861075459bcd68e870551241c13dd92f623fdc Nov 03 01:28:09.113939 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:09.113916 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-h9gfn"] Nov 03 01:28:09.436105 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:09.436015 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-h9gfn" event={"ID":"d4d4e0c0-2ae5-402d-a47a-d8e4c79d7cfa","Type":"ContainerStarted","Data":"3f05b45878b69de1d144b0f7cc861075459bcd68e870551241c13dd92f623fdc"} Nov 03 01:28:10.442679 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:10.442638 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-h9gfn" event={"ID":"d4d4e0c0-2ae5-402d-a47a-d8e4c79d7cfa","Type":"ContainerStarted","Data":"f8bc4cad99a617c81d2100928caf2d733c577d5b854f77f68bb07d5ca76e947f"} Nov 03 01:28:10.564647 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:10.564599 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-h9gfn" podStartSLOduration=1.960259102 podStartE2EDuration="2.564586076s" totalImagesPullingTime="604.326974ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:28:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:28:10.563804043 +0000 UTC m=+572.593286338" watchObservedRunningTime="2026-11-03 01:28:10.564586076 +0000 UTC m=+572.594068371" Nov 03 01:28:10.747262 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:10.747179 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-h9gfn" Nov 03 01:28:12.405788 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:12.405747 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qrnjn" Nov 03 01:28:12.418607 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:12.418584 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qrnjn" Nov 03 01:28:12.463236 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:12.463204 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qrnjn" Nov 03 01:28:12.790296 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:12.790211 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l87z7"] Nov 03 01:28:12.793770 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:12.793750 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l87z7" Nov 03 01:28:12.850110 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:12.850082 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Nov 03 01:28:12.851583 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:12.851556 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l87z7"] Nov 03 01:28:12.955973 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:12.955936 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:12.975960 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:12.975925 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03a82c52-52db-44b7-bf71-64ec0c963054-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-l87z7\" (UID: \"03a82c52-52db-44b7-bf71-64ec0c963054\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l87z7" Nov 03 01:28:12.976128 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:12.976061 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03a82c52-52db-44b7-bf71-64ec0c963054-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-l87z7\" (UID: \"03a82c52-52db-44b7-bf71-64ec0c963054\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l87z7" Nov 03 01:28:12.976128 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:12.976105 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03a82c52-52db-44b7-bf71-64ec0c963054-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-l87z7\" (UID: \"03a82c52-52db-44b7-bf71-64ec0c963054\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l87z7" Nov 03 01:28:12.976243 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:12.976141 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmgc2\" (UniqueName: \"kubernetes.io/projected/03a82c52-52db-44b7-bf71-64ec0c963054-kube-api-access-fmgc2\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-l87z7\" (UID: \"03a82c52-52db-44b7-bf71-64ec0c963054\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l87z7" Nov 03 01:28:12.976243 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:12.976203 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03a82c52-52db-44b7-bf71-64ec0c963054-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-l87z7\" (UID: \"03a82c52-52db-44b7-bf71-64ec0c963054\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l87z7" Nov 03 01:28:12.976326 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:12.976295 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03a82c52-52db-44b7-bf71-64ec0c963054-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-l87z7\" (UID: \"03a82c52-52db-44b7-bf71-64ec0c963054\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l87z7" Nov 03 01:28:13.077263 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:13.077217 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03a82c52-52db-44b7-bf71-64ec0c963054-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-l87z7\" (UID: \"03a82c52-52db-44b7-bf71-64ec0c963054\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l87z7" Nov 03 01:28:13.077484 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:13.077322 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03a82c52-52db-44b7-bf71-64ec0c963054-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-l87z7\" (UID: \"03a82c52-52db-44b7-bf71-64ec0c963054\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l87z7" Nov 03 01:28:13.077484 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:13.077375 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03a82c52-52db-44b7-bf71-64ec0c963054-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-l87z7\" (UID: \"03a82c52-52db-44b7-bf71-64ec0c963054\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l87z7" Nov 03 01:28:13.077484 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:13.077417 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmgc2\" (UniqueName: \"kubernetes.io/projected/03a82c52-52db-44b7-bf71-64ec0c963054-kube-api-access-fmgc2\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-l87z7\" (UID: \"03a82c52-52db-44b7-bf71-64ec0c963054\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l87z7" Nov 03 01:28:13.077484 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:13.077453 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03a82c52-52db-44b7-bf71-64ec0c963054-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-l87z7\" (UID: \"03a82c52-52db-44b7-bf71-64ec0c963054\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l87z7" Nov 03 01:28:13.077484 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:13.077480 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03a82c52-52db-44b7-bf71-64ec0c963054-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-l87z7\" (UID: \"03a82c52-52db-44b7-bf71-64ec0c963054\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l87z7" Nov 03 01:28:13.077802 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:13.077772 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03a82c52-52db-44b7-bf71-64ec0c963054-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-l87z7\" (UID: \"03a82c52-52db-44b7-bf71-64ec0c963054\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l87z7" Nov 03 01:28:13.077870 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:13.077805 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03a82c52-52db-44b7-bf71-64ec0c963054-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-l87z7\" (UID: \"03a82c52-52db-44b7-bf71-64ec0c963054\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l87z7" Nov 03 01:28:13.077870 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:13.077841 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03a82c52-52db-44b7-bf71-64ec0c963054-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-l87z7\" (UID: \"03a82c52-52db-44b7-bf71-64ec0c963054\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l87z7" Nov 03 01:28:13.079643 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:13.079620 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03a82c52-52db-44b7-bf71-64ec0c963054-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-l87z7\" (UID: \"03a82c52-52db-44b7-bf71-64ec0c963054\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l87z7" Nov 03 01:28:13.080028 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:13.080006 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03a82c52-52db-44b7-bf71-64ec0c963054-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-l87z7\" (UID: \"03a82c52-52db-44b7-bf71-64ec0c963054\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l87z7" Nov 03 01:28:13.140224 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:13.140184 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmgc2\" (UniqueName: \"kubernetes.io/projected/03a82c52-52db-44b7-bf71-64ec0c963054-kube-api-access-fmgc2\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-l87z7\" (UID: \"03a82c52-52db-44b7-bf71-64ec0c963054\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l87z7" Nov 03 01:28:13.421703 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:13.421616 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l87z7" Nov 03 01:28:13.792456 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:28:13.792415 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod03a82c52_52db_44b7_bf71_64ec0c963054.slice/crio-4f9a721cf77780df94b21a046405a8a09735018b1d86cf947fac480c208e5fbe WatchSource:0}: Error finding container 4f9a721cf77780df94b21a046405a8a09735018b1d86cf947fac480c208e5fbe: Status 404 returned error can't find the container with id 4f9a721cf77780df94b21a046405a8a09735018b1d86cf947fac480c208e5fbe Nov 03 01:28:13.794618 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:13.794582 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l87z7"] Nov 03 01:28:14.035745 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:28:14.035704 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 01:28:14.468851 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:14.468805 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l87z7" event={"ID":"03a82c52-52db-44b7-bf71-64ec0c963054","Type":"ContainerStarted","Data":"0cb6d564ef37a81a0ccd1080b836c6ae6d4f73dc8063587cc51a78175273fc35"} Nov 03 01:28:14.469276 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:14.468872 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l87z7" event={"ID":"03a82c52-52db-44b7-bf71-64ec0c963054","Type":"ContainerStarted","Data":"4f9a721cf77780df94b21a046405a8a09735018b1d86cf947fac480c208e5fbe"} Nov 03 01:28:14.590777 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:14.590714 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l87z7" podStartSLOduration=2.000536758 podStartE2EDuration="2.59070123s" totalImagesPullingTime="590.164472ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:28:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:28:14.590234824 +0000 UTC m=+576.619717123" watchObservedRunningTime="2026-11-03 01:28:14.59070123 +0000 UTC m=+576.620183522" Nov 03 01:28:15.422626 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:15.422575 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l87z7" Nov 03 01:28:17.386791 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:17.386751 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n8wz8"] Nov 03 01:28:17.393067 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:17.393038 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n8wz8" Nov 03 01:28:17.419180 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:17.419126 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hfxj\" (UniqueName: \"kubernetes.io/projected/9b955a41-8e08-46b0-87d0-64ab0cc1e381-kube-api-access-6hfxj\") pod \"maas-default-gateway-openshift-default-59f957fdd7-n8wz8\" (UID: \"9b955a41-8e08-46b0-87d0-64ab0cc1e381\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n8wz8" Nov 03 01:28:17.419392 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:17.419194 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b955a41-8e08-46b0-87d0-64ab0cc1e381-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-n8wz8\" (UID: \"9b955a41-8e08-46b0-87d0-64ab0cc1e381\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n8wz8" Nov 03 01:28:17.419392 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:17.419315 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b955a41-8e08-46b0-87d0-64ab0cc1e381-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-n8wz8\" (UID: \"9b955a41-8e08-46b0-87d0-64ab0cc1e381\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n8wz8" Nov 03 01:28:17.419392 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:17.419353 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b955a41-8e08-46b0-87d0-64ab0cc1e381-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-n8wz8\" (UID: \"9b955a41-8e08-46b0-87d0-64ab0cc1e381\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n8wz8" Nov 03 01:28:17.419505 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:17.419472 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b955a41-8e08-46b0-87d0-64ab0cc1e381-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-n8wz8\" (UID: \"9b955a41-8e08-46b0-87d0-64ab0cc1e381\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n8wz8" Nov 03 01:28:17.419505 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:17.419499 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b955a41-8e08-46b0-87d0-64ab0cc1e381-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-n8wz8\" (UID: \"9b955a41-8e08-46b0-87d0-64ab0cc1e381\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n8wz8" Nov 03 01:28:17.419569 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:17.419517 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b955a41-8e08-46b0-87d0-64ab0cc1e381-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-n8wz8\" (UID: \"9b955a41-8e08-46b0-87d0-64ab0cc1e381\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n8wz8" Nov 03 01:28:17.419569 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:17.419533 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b955a41-8e08-46b0-87d0-64ab0cc1e381-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-n8wz8\" (UID: \"9b955a41-8e08-46b0-87d0-64ab0cc1e381\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n8wz8" Nov 03 01:28:17.419628 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:17.419612 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b955a41-8e08-46b0-87d0-64ab0cc1e381-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-n8wz8\" (UID: \"9b955a41-8e08-46b0-87d0-64ab0cc1e381\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n8wz8" Nov 03 01:28:17.447481 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:17.447439 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-w2lkg\"" Nov 03 01:28:17.447481 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:17.447439 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 03 01:28:17.514957 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:17.514920 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n8wz8"] Nov 03 01:28:17.520461 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:17.520422 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b955a41-8e08-46b0-87d0-64ab0cc1e381-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-n8wz8\" (UID: \"9b955a41-8e08-46b0-87d0-64ab0cc1e381\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n8wz8" Nov 03 01:28:17.520461 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:17.520464 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b955a41-8e08-46b0-87d0-64ab0cc1e381-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-n8wz8\" (UID: \"9b955a41-8e08-46b0-87d0-64ab0cc1e381\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n8wz8" Nov 03 01:28:17.520730 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:17.520598 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b955a41-8e08-46b0-87d0-64ab0cc1e381-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-n8wz8\" (UID: \"9b955a41-8e08-46b0-87d0-64ab0cc1e381\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n8wz8" Nov 03 01:28:17.520730 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:17.520648 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b955a41-8e08-46b0-87d0-64ab0cc1e381-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-n8wz8\" (UID: \"9b955a41-8e08-46b0-87d0-64ab0cc1e381\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n8wz8" Nov 03 01:28:17.520730 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:17.520671 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b955a41-8e08-46b0-87d0-64ab0cc1e381-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-n8wz8\" (UID: \"9b955a41-8e08-46b0-87d0-64ab0cc1e381\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n8wz8" Nov 03 01:28:17.520730 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:17.520699 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b955a41-8e08-46b0-87d0-64ab0cc1e381-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-n8wz8\" (UID: \"9b955a41-8e08-46b0-87d0-64ab0cc1e381\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n8wz8" Nov 03 01:28:17.520935 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:17.520740 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b955a41-8e08-46b0-87d0-64ab0cc1e381-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-n8wz8\" (UID: \"9b955a41-8e08-46b0-87d0-64ab0cc1e381\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n8wz8" Nov 03 01:28:17.520935 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:17.520778 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hfxj\" (UniqueName: \"kubernetes.io/projected/9b955a41-8e08-46b0-87d0-64ab0cc1e381-kube-api-access-6hfxj\") pod \"maas-default-gateway-openshift-default-59f957fdd7-n8wz8\" (UID: \"9b955a41-8e08-46b0-87d0-64ab0cc1e381\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n8wz8" Nov 03 01:28:17.520935 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:17.520857 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b955a41-8e08-46b0-87d0-64ab0cc1e381-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-n8wz8\" (UID: \"9b955a41-8e08-46b0-87d0-64ab0cc1e381\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n8wz8" Nov 03 01:28:17.521236 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:17.521217 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b955a41-8e08-46b0-87d0-64ab0cc1e381-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-n8wz8\" (UID: \"9b955a41-8e08-46b0-87d0-64ab0cc1e381\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n8wz8" Nov 03 01:28:17.521540 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:17.521512 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b955a41-8e08-46b0-87d0-64ab0cc1e381-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-n8wz8\" (UID: \"9b955a41-8e08-46b0-87d0-64ab0cc1e381\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n8wz8" Nov 03 01:28:17.521660 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:17.521546 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b955a41-8e08-46b0-87d0-64ab0cc1e381-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-n8wz8\" (UID: \"9b955a41-8e08-46b0-87d0-64ab0cc1e381\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n8wz8" Nov 03 01:28:17.521660 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:17.521631 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b955a41-8e08-46b0-87d0-64ab0cc1e381-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-n8wz8\" (UID: \"9b955a41-8e08-46b0-87d0-64ab0cc1e381\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n8wz8" Nov 03 01:28:17.521780 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:17.521695 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b955a41-8e08-46b0-87d0-64ab0cc1e381-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-n8wz8\" (UID: \"9b955a41-8e08-46b0-87d0-64ab0cc1e381\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n8wz8" Nov 03 01:28:17.523385 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:17.523333 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b955a41-8e08-46b0-87d0-64ab0cc1e381-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-n8wz8\" (UID: \"9b955a41-8e08-46b0-87d0-64ab0cc1e381\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n8wz8" Nov 03 01:28:17.523545 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:17.523521 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b955a41-8e08-46b0-87d0-64ab0cc1e381-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-n8wz8\" (UID: \"9b955a41-8e08-46b0-87d0-64ab0cc1e381\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n8wz8" Nov 03 01:28:17.583969 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:17.583927 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b955a41-8e08-46b0-87d0-64ab0cc1e381-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-n8wz8\" (UID: \"9b955a41-8e08-46b0-87d0-64ab0cc1e381\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n8wz8" Nov 03 01:28:17.584124 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:17.584067 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hfxj\" (UniqueName: \"kubernetes.io/projected/9b955a41-8e08-46b0-87d0-64ab0cc1e381-kube-api-access-6hfxj\") pod \"maas-default-gateway-openshift-default-59f957fdd7-n8wz8\" (UID: \"9b955a41-8e08-46b0-87d0-64ab0cc1e381\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n8wz8" Nov 03 01:28:17.709280 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:17.709184 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n8wz8" Nov 03 01:28:18.127992 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:28:18.127957 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b955a41_8e08_46b0_87d0_64ab0cc1e381.slice/crio-b8e87c34169361b3c79991b7273a47c0e4870299fd73ec95cd50d555d3cc9d51 WatchSource:0}: Error finding container b8e87c34169361b3c79991b7273a47c0e4870299fd73ec95cd50d555d3cc9d51: Status 404 returned error can't find the container with id b8e87c34169361b3c79991b7273a47c0e4870299fd73ec95cd50d555d3cc9d51 Nov 03 01:28:18.130164 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:18.130140 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n8wz8"] Nov 03 01:28:18.494562 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:18.494458 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n8wz8" event={"ID":"9b955a41-8e08-46b0-87d0-64ab0cc1e381","Type":"ContainerStarted","Data":"b8e87c34169361b3c79991b7273a47c0e4870299fd73ec95cd50d555d3cc9d51"} Nov 03 01:28:18.747095 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:18.746996 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-h9gfn" Nov 03 01:28:18.759832 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:18.759799 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-h9gfn" Nov 03 01:28:19.510275 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:19.510243 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-h9gfn" Nov 03 01:28:22.979461 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:22.974073 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 01:28:22.979461 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:22.974253 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 01:28:22.979461 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:22.974341 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 01:28:23.422501 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:23.422456 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l87z7" Nov 03 01:28:23.437619 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:23.437580 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l87z7" Nov 03 01:28:23.522661 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:23.522624 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n8wz8" event={"ID":"9b955a41-8e08-46b0-87d0-64ab0cc1e381","Type":"ContainerStarted","Data":"3511ea5db74e3c69162bea6aeac9babd9d71c73b9d9d163be8902526583026c4"} Nov 03 01:28:23.542684 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:23.542651 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l87z7" Nov 03 01:28:23.686496 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:23.686383 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n8wz8" podStartSLOduration=1.843788048 podStartE2EDuration="6.686347652s" totalImagesPullingTime="4.842559604s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:28:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:28:23.685415437 +0000 UTC m=+585.714897732" watchObservedRunningTime="2026-11-03 01:28:23.686347652 +0000 UTC m=+585.715830009" Nov 03 01:28:23.709523 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:23.709479 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n8wz8" Nov 03 01:28:24.711148 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:24.711113 5437 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-n8wz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Nov 03 01:28:24.711583 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:24.711175 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n8wz8" podUID="9b955a41-8e08-46b0-87d0-64ab0cc1e381" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Nov 03 01:28:25.710475 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:25.710436 5437 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-n8wz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Nov 03 01:28:25.710670 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:25.710508 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n8wz8" podUID="9b955a41-8e08-46b0-87d0-64ab0cc1e381" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Nov 03 01:28:26.709803 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:26.709770 5437 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-n8wz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Nov 03 01:28:26.710222 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:26.709836 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n8wz8" podUID="9b955a41-8e08-46b0-87d0-64ab0cc1e381" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Nov 03 01:28:27.710316 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:27.710273 5437 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-n8wz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Nov 03 01:28:27.710752 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:27.710340 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n8wz8" podUID="9b955a41-8e08-46b0-87d0-64ab0cc1e381" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Nov 03 01:28:28.710001 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:28.709958 5437 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-n8wz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Nov 03 01:28:28.710196 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:28.710021 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n8wz8" podUID="9b955a41-8e08-46b0-87d0-64ab0cc1e381" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Nov 03 01:28:29.709615 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:29.709574 5437 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-n8wz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Nov 03 01:28:29.710045 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:29.709638 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n8wz8" podUID="9b955a41-8e08-46b0-87d0-64ab0cc1e381" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Nov 03 01:28:31.513430 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:31.513393 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n8wz8" Nov 03 01:28:31.513898 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:31.513763 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n8wz8" Nov 03 01:28:31.514601 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:31.514577 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n8wz8" Nov 03 01:28:32.504670 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:32.504632 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-x2xwt"] Nov 03 01:28:32.508851 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:32.508828 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-x2xwt" Nov 03 01:28:32.592098 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:32.592051 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9cb50c6-d76d-4138-8d00-6b7836297ed7-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-x2xwt\" (UID: \"f9cb50c6-d76d-4138-8d00-6b7836297ed7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-x2xwt" Nov 03 01:28:32.592584 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:32.592166 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9cb50c6-d76d-4138-8d00-6b7836297ed7-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-x2xwt\" (UID: \"f9cb50c6-d76d-4138-8d00-6b7836297ed7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-x2xwt" Nov 03 01:28:32.592584 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:32.592238 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9cb50c6-d76d-4138-8d00-6b7836297ed7-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-x2xwt\" (UID: \"f9cb50c6-d76d-4138-8d00-6b7836297ed7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-x2xwt" Nov 03 01:28:32.592584 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:32.592303 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9cb50c6-d76d-4138-8d00-6b7836297ed7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-x2xwt\" (UID: \"f9cb50c6-d76d-4138-8d00-6b7836297ed7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-x2xwt" Nov 03 01:28:32.592584 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:32.592414 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9cb50c6-d76d-4138-8d00-6b7836297ed7-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-x2xwt\" (UID: \"f9cb50c6-d76d-4138-8d00-6b7836297ed7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-x2xwt" Nov 03 01:28:32.592584 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:32.592440 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvr4k\" (UniqueName: \"kubernetes.io/projected/f9cb50c6-d76d-4138-8d00-6b7836297ed7-kube-api-access-wvr4k\") pod \"maas-default-gateway-openshift-default-59f957fdd7-x2xwt\" (UID: \"f9cb50c6-d76d-4138-8d00-6b7836297ed7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-x2xwt" Nov 03 01:28:32.592584 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:32.592493 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9cb50c6-d76d-4138-8d00-6b7836297ed7-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-x2xwt\" (UID: \"f9cb50c6-d76d-4138-8d00-6b7836297ed7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-x2xwt" Nov 03 01:28:32.592584 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:32.592552 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9cb50c6-d76d-4138-8d00-6b7836297ed7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-x2xwt\" (UID: \"f9cb50c6-d76d-4138-8d00-6b7836297ed7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-x2xwt" Nov 03 01:28:32.592864 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:32.592601 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9cb50c6-d76d-4138-8d00-6b7836297ed7-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-x2xwt\" (UID: \"f9cb50c6-d76d-4138-8d00-6b7836297ed7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-x2xwt" Nov 03 01:28:32.596087 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:32.596060 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-x2xwt"] Nov 03 01:28:32.693674 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:32.693631 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9cb50c6-d76d-4138-8d00-6b7836297ed7-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-x2xwt\" (UID: \"f9cb50c6-d76d-4138-8d00-6b7836297ed7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-x2xwt" Nov 03 01:28:32.694083 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:32.694042 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvr4k\" (UniqueName: \"kubernetes.io/projected/f9cb50c6-d76d-4138-8d00-6b7836297ed7-kube-api-access-wvr4k\") pod \"maas-default-gateway-openshift-default-59f957fdd7-x2xwt\" (UID: \"f9cb50c6-d76d-4138-8d00-6b7836297ed7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-x2xwt" Nov 03 01:28:32.694215 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:32.694092 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9cb50c6-d76d-4138-8d00-6b7836297ed7-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-x2xwt\" (UID: \"f9cb50c6-d76d-4138-8d00-6b7836297ed7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-x2xwt" Nov 03 01:28:32.694215 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:32.694117 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9cb50c6-d76d-4138-8d00-6b7836297ed7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-x2xwt\" (UID: \"f9cb50c6-d76d-4138-8d00-6b7836297ed7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-x2xwt" Nov 03 01:28:32.694215 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:32.694149 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9cb50c6-d76d-4138-8d00-6b7836297ed7-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-x2xwt\" (UID: \"f9cb50c6-d76d-4138-8d00-6b7836297ed7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-x2xwt" Nov 03 01:28:32.694215 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:32.694177 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9cb50c6-d76d-4138-8d00-6b7836297ed7-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-x2xwt\" (UID: \"f9cb50c6-d76d-4138-8d00-6b7836297ed7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-x2xwt" Nov 03 01:28:32.694215 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:32.694200 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9cb50c6-d76d-4138-8d00-6b7836297ed7-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-x2xwt\" (UID: \"f9cb50c6-d76d-4138-8d00-6b7836297ed7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-x2xwt" Nov 03 01:28:32.694421 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:32.694249 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9cb50c6-d76d-4138-8d00-6b7836297ed7-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-x2xwt\" (UID: \"f9cb50c6-d76d-4138-8d00-6b7836297ed7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-x2xwt" Nov 03 01:28:32.694421 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:32.694325 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9cb50c6-d76d-4138-8d00-6b7836297ed7-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-x2xwt\" (UID: \"f9cb50c6-d76d-4138-8d00-6b7836297ed7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-x2xwt" Nov 03 01:28:32.694421 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:32.694388 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9cb50c6-d76d-4138-8d00-6b7836297ed7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-x2xwt\" (UID: \"f9cb50c6-d76d-4138-8d00-6b7836297ed7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-x2xwt" Nov 03 01:28:32.694421 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:32.694394 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9cb50c6-d76d-4138-8d00-6b7836297ed7-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-x2xwt\" (UID: \"f9cb50c6-d76d-4138-8d00-6b7836297ed7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-x2xwt" Nov 03 01:28:32.694904 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:32.694874 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9cb50c6-d76d-4138-8d00-6b7836297ed7-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-x2xwt\" (UID: \"f9cb50c6-d76d-4138-8d00-6b7836297ed7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-x2xwt" Nov 03 01:28:32.695029 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:32.694958 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9cb50c6-d76d-4138-8d00-6b7836297ed7-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-x2xwt\" (UID: \"f9cb50c6-d76d-4138-8d00-6b7836297ed7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-x2xwt" Nov 03 01:28:32.695090 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:32.695045 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9cb50c6-d76d-4138-8d00-6b7836297ed7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-x2xwt\" (UID: \"f9cb50c6-d76d-4138-8d00-6b7836297ed7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-x2xwt" Nov 03 01:28:32.697023 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:32.696995 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9cb50c6-d76d-4138-8d00-6b7836297ed7-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-x2xwt\" (UID: \"f9cb50c6-d76d-4138-8d00-6b7836297ed7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-x2xwt" Nov 03 01:28:32.697204 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:32.697183 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9cb50c6-d76d-4138-8d00-6b7836297ed7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-x2xwt\" (UID: \"f9cb50c6-d76d-4138-8d00-6b7836297ed7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-x2xwt" Nov 03 01:28:32.757137 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:32.757048 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9cb50c6-d76d-4138-8d00-6b7836297ed7-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-x2xwt\" (UID: \"f9cb50c6-d76d-4138-8d00-6b7836297ed7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-x2xwt" Nov 03 01:28:32.757306 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:32.757146 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvr4k\" (UniqueName: \"kubernetes.io/projected/f9cb50c6-d76d-4138-8d00-6b7836297ed7-kube-api-access-wvr4k\") pod \"maas-default-gateway-openshift-default-59f957fdd7-x2xwt\" (UID: \"f9cb50c6-d76d-4138-8d00-6b7836297ed7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-x2xwt" Nov 03 01:28:32.823524 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:32.823483 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-x2xwt" Nov 03 01:28:33.243250 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:28:33.243217 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9cb50c6_d76d_4138_8d00_6b7836297ed7.slice/crio-5c3c7d97b87a8b2987f00bca1015a4aa7942fcde5683206b1f7d3fdd7f5b5158 WatchSource:0}: Error finding container 5c3c7d97b87a8b2987f00bca1015a4aa7942fcde5683206b1f7d3fdd7f5b5158: Status 404 returned error can't find the container with id 5c3c7d97b87a8b2987f00bca1015a4aa7942fcde5683206b1f7d3fdd7f5b5158 Nov 03 01:28:33.244619 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:33.244592 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-x2xwt"] Nov 03 01:28:33.246841 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:33.246809 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 01:28:33.246933 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:33.246876 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 01:28:33.246933 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:33.246908 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 01:28:33.597994 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:33.597953 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-x2xwt" event={"ID":"f9cb50c6-d76d-4138-8d00-6b7836297ed7","Type":"ContainerStarted","Data":"9f644017f3f0d4da9def90eadb30e0ac54f0703198433d0380c3ad26f51ab5ef"} Nov 03 01:28:33.598564 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:33.598534 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-x2xwt" event={"ID":"f9cb50c6-d76d-4138-8d00-6b7836297ed7","Type":"ContainerStarted","Data":"5c3c7d97b87a8b2987f00bca1015a4aa7942fcde5683206b1f7d3fdd7f5b5158"} Nov 03 01:28:33.723260 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:33.723187 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-x2xwt" podStartSLOduration=1.723165059 podStartE2EDuration="1.723165059s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:28:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:28:33.721824455 +0000 UTC m=+595.751306750" watchObservedRunningTime="2026-11-03 01:28:33.723165059 +0000 UTC m=+595.752647355" Nov 03 01:28:33.831525 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:33.831475 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-x2xwt" Nov 03 01:28:34.834483 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:34.834442 5437 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-x2xwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 03 01:28:34.835382 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:34.835056 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-x2xwt" podUID="f9cb50c6-d76d-4138-8d00-6b7836297ed7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 03 01:28:35.839389 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:35.838855 5437 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-x2xwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 03 01:28:35.839389 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:35.838933 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-x2xwt" podUID="f9cb50c6-d76d-4138-8d00-6b7836297ed7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 03 01:28:36.830843 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:36.830800 5437 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-x2xwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 03 01:28:36.831160 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:36.831125 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-x2xwt" podUID="f9cb50c6-d76d-4138-8d00-6b7836297ed7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 03 01:28:37.831824 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:37.831780 5437 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-x2xwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 03 01:28:37.832442 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:37.832405 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-x2xwt" podUID="f9cb50c6-d76d-4138-8d00-6b7836297ed7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 03 01:28:38.836877 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:38.836834 5437 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-x2xwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 03 01:28:38.837529 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:38.837488 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-x2xwt" podUID="f9cb50c6-d76d-4138-8d00-6b7836297ed7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 03 01:28:38.986696 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:38.986658 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:39.824164 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:39.824126 5437 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-x2xwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 03 01:28:39.824437 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:39.824191 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-x2xwt" podUID="f9cb50c6-d76d-4138-8d00-6b7836297ed7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 03 01:28:40.824805 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:40.824764 5437 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-x2xwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 03 01:28:40.825212 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:40.824828 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-x2xwt" podUID="f9cb50c6-d76d-4138-8d00-6b7836297ed7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 03 01:28:41.293618 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:41.293516 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:41.497560 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:41.497521 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:41.831215 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:41.831171 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-x2xwt" Nov 03 01:28:41.831935 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:41.831905 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-x2xwt" Nov 03 01:28:41.832487 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:41.832468 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-x2xwt" Nov 03 01:28:42.906282 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:42.906239 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:43.117527 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:28:43.117491 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:44.034509 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:28:44.034466 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 01:29:11.683751 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:11.683719 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:11.894527 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:11.894475 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:14.035862 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:29:14.035828 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 01:29:18.198522 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:18.198429 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:18.455629 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:18.455506 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:18.785060 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:18.784966 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:19.042810 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:19.042714 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:19.261530 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:19.261491 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:19.464606 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:19.464573 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:19.524227 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:19.524183 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-e3f0a557-gw-openshift-default-7d94d6b5f7-fts5d"] Nov 03 01:29:19.527148 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:19.527123 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-e3f0a557-gw-openshift-default-7d94d6b5f7-fts5d" Nov 03 01:29:19.585006 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:19.584561 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-e3f0a557-gw-openshift-default-dockercfg-9fh72\"" Nov 03 01:29:19.606257 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:19.606222 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-e3f0a557-gw-openshift-default-7d94d6b5f7-fts5d"] Nov 03 01:29:19.672916 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:19.672873 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:19.687015 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:19.686967 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b980620-e583-42e5-b141-264c2b72bfb2-istio-envoy\") pod \"e2e-migrate-e3f0a557-gw-openshift-default-7d94d6b5f7-fts5d\" (UID: \"3b980620-e583-42e5-b141-264c2b72bfb2\") " pod="openshift-ingress/e2e-migrate-e3f0a557-gw-openshift-default-7d94d6b5f7-fts5d" Nov 03 01:29:19.687221 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:19.687081 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b980620-e583-42e5-b141-264c2b72bfb2-istio-token\") pod \"e2e-migrate-e3f0a557-gw-openshift-default-7d94d6b5f7-fts5d\" (UID: \"3b980620-e583-42e5-b141-264c2b72bfb2\") " pod="openshift-ingress/e2e-migrate-e3f0a557-gw-openshift-default-7d94d6b5f7-fts5d" Nov 03 01:29:19.687221 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:19.687144 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b980620-e583-42e5-b141-264c2b72bfb2-istio-data\") pod \"e2e-migrate-e3f0a557-gw-openshift-default-7d94d6b5f7-fts5d\" (UID: \"3b980620-e583-42e5-b141-264c2b72bfb2\") " pod="openshift-ingress/e2e-migrate-e3f0a557-gw-openshift-default-7d94d6b5f7-fts5d" Nov 03 01:29:19.687359 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:19.687265 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b980620-e583-42e5-b141-264c2b72bfb2-istiod-ca-cert\") pod \"e2e-migrate-e3f0a557-gw-openshift-default-7d94d6b5f7-fts5d\" (UID: \"3b980620-e583-42e5-b141-264c2b72bfb2\") " pod="openshift-ingress/e2e-migrate-e3f0a557-gw-openshift-default-7d94d6b5f7-fts5d" Nov 03 01:29:19.687359 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:19.687300 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b980620-e583-42e5-b141-264c2b72bfb2-credential-socket\") pod \"e2e-migrate-e3f0a557-gw-openshift-default-7d94d6b5f7-fts5d\" (UID: \"3b980620-e583-42e5-b141-264c2b72bfb2\") " pod="openshift-ingress/e2e-migrate-e3f0a557-gw-openshift-default-7d94d6b5f7-fts5d" Nov 03 01:29:19.687482 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:19.687406 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b980620-e583-42e5-b141-264c2b72bfb2-workload-certs\") pod \"e2e-migrate-e3f0a557-gw-openshift-default-7d94d6b5f7-fts5d\" (UID: \"3b980620-e583-42e5-b141-264c2b72bfb2\") " pod="openshift-ingress/e2e-migrate-e3f0a557-gw-openshift-default-7d94d6b5f7-fts5d" Nov 03 01:29:19.687482 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:19.687440 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48bmb\" (UniqueName: \"kubernetes.io/projected/3b980620-e583-42e5-b141-264c2b72bfb2-kube-api-access-48bmb\") pod \"e2e-migrate-e3f0a557-gw-openshift-default-7d94d6b5f7-fts5d\" (UID: \"3b980620-e583-42e5-b141-264c2b72bfb2\") " pod="openshift-ingress/e2e-migrate-e3f0a557-gw-openshift-default-7d94d6b5f7-fts5d" Nov 03 01:29:19.687565 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:19.687514 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b980620-e583-42e5-b141-264c2b72bfb2-workload-socket\") pod \"e2e-migrate-e3f0a557-gw-openshift-default-7d94d6b5f7-fts5d\" (UID: \"3b980620-e583-42e5-b141-264c2b72bfb2\") " pod="openshift-ingress/e2e-migrate-e3f0a557-gw-openshift-default-7d94d6b5f7-fts5d" Nov 03 01:29:19.687565 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:19.687537 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b980620-e583-42e5-b141-264c2b72bfb2-istio-podinfo\") pod \"e2e-migrate-e3f0a557-gw-openshift-default-7d94d6b5f7-fts5d\" (UID: \"3b980620-e583-42e5-b141-264c2b72bfb2\") " pod="openshift-ingress/e2e-migrate-e3f0a557-gw-openshift-default-7d94d6b5f7-fts5d" Nov 03 01:29:19.788354 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:19.788256 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b980620-e583-42e5-b141-264c2b72bfb2-istio-data\") pod \"e2e-migrate-e3f0a557-gw-openshift-default-7d94d6b5f7-fts5d\" (UID: \"3b980620-e583-42e5-b141-264c2b72bfb2\") " pod="openshift-ingress/e2e-migrate-e3f0a557-gw-openshift-default-7d94d6b5f7-fts5d" Nov 03 01:29:19.788354 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:19.788310 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b980620-e583-42e5-b141-264c2b72bfb2-istiod-ca-cert\") pod \"e2e-migrate-e3f0a557-gw-openshift-default-7d94d6b5f7-fts5d\" (UID: \"3b980620-e583-42e5-b141-264c2b72bfb2\") " pod="openshift-ingress/e2e-migrate-e3f0a557-gw-openshift-default-7d94d6b5f7-fts5d" Nov 03 01:29:19.788354 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:19.788337 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b980620-e583-42e5-b141-264c2b72bfb2-credential-socket\") pod \"e2e-migrate-e3f0a557-gw-openshift-default-7d94d6b5f7-fts5d\" (UID: \"3b980620-e583-42e5-b141-264c2b72bfb2\") " pod="openshift-ingress/e2e-migrate-e3f0a557-gw-openshift-default-7d94d6b5f7-fts5d" Nov 03 01:29:19.788674 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:19.788397 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b980620-e583-42e5-b141-264c2b72bfb2-workload-certs\") pod \"e2e-migrate-e3f0a557-gw-openshift-default-7d94d6b5f7-fts5d\" (UID: \"3b980620-e583-42e5-b141-264c2b72bfb2\") " pod="openshift-ingress/e2e-migrate-e3f0a557-gw-openshift-default-7d94d6b5f7-fts5d" Nov 03 01:29:19.788674 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:19.788431 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48bmb\" (UniqueName: \"kubernetes.io/projected/3b980620-e583-42e5-b141-264c2b72bfb2-kube-api-access-48bmb\") pod \"e2e-migrate-e3f0a557-gw-openshift-default-7d94d6b5f7-fts5d\" (UID: \"3b980620-e583-42e5-b141-264c2b72bfb2\") " pod="openshift-ingress/e2e-migrate-e3f0a557-gw-openshift-default-7d94d6b5f7-fts5d" Nov 03 01:29:19.788674 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:19.788486 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b980620-e583-42e5-b141-264c2b72bfb2-workload-socket\") pod \"e2e-migrate-e3f0a557-gw-openshift-default-7d94d6b5f7-fts5d\" (UID: \"3b980620-e583-42e5-b141-264c2b72bfb2\") " pod="openshift-ingress/e2e-migrate-e3f0a557-gw-openshift-default-7d94d6b5f7-fts5d" Nov 03 01:29:19.788674 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:19.788508 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b980620-e583-42e5-b141-264c2b72bfb2-istio-podinfo\") pod \"e2e-migrate-e3f0a557-gw-openshift-default-7d94d6b5f7-fts5d\" (UID: \"3b980620-e583-42e5-b141-264c2b72bfb2\") " pod="openshift-ingress/e2e-migrate-e3f0a557-gw-openshift-default-7d94d6b5f7-fts5d" Nov 03 01:29:19.788674 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:19.788568 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b980620-e583-42e5-b141-264c2b72bfb2-istio-envoy\") pod \"e2e-migrate-e3f0a557-gw-openshift-default-7d94d6b5f7-fts5d\" (UID: \"3b980620-e583-42e5-b141-264c2b72bfb2\") " pod="openshift-ingress/e2e-migrate-e3f0a557-gw-openshift-default-7d94d6b5f7-fts5d" Nov 03 01:29:19.788674 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:19.788605 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b980620-e583-42e5-b141-264c2b72bfb2-istio-token\") pod \"e2e-migrate-e3f0a557-gw-openshift-default-7d94d6b5f7-fts5d\" (UID: \"3b980620-e583-42e5-b141-264c2b72bfb2\") " pod="openshift-ingress/e2e-migrate-e3f0a557-gw-openshift-default-7d94d6b5f7-fts5d" Nov 03 01:29:19.788966 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:19.788775 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b980620-e583-42e5-b141-264c2b72bfb2-istio-data\") pod \"e2e-migrate-e3f0a557-gw-openshift-default-7d94d6b5f7-fts5d\" (UID: \"3b980620-e583-42e5-b141-264c2b72bfb2\") " pod="openshift-ingress/e2e-migrate-e3f0a557-gw-openshift-default-7d94d6b5f7-fts5d" Nov 03 01:29:19.788966 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:19.788792 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b980620-e583-42e5-b141-264c2b72bfb2-credential-socket\") pod \"e2e-migrate-e3f0a557-gw-openshift-default-7d94d6b5f7-fts5d\" (UID: \"3b980620-e583-42e5-b141-264c2b72bfb2\") " pod="openshift-ingress/e2e-migrate-e3f0a557-gw-openshift-default-7d94d6b5f7-fts5d" Nov 03 01:29:19.788966 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:19.788912 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b980620-e583-42e5-b141-264c2b72bfb2-workload-certs\") pod \"e2e-migrate-e3f0a557-gw-openshift-default-7d94d6b5f7-fts5d\" (UID: \"3b980620-e583-42e5-b141-264c2b72bfb2\") " pod="openshift-ingress/e2e-migrate-e3f0a557-gw-openshift-default-7d94d6b5f7-fts5d" Nov 03 01:29:19.789125 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:19.789059 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b980620-e583-42e5-b141-264c2b72bfb2-workload-socket\") pod \"e2e-migrate-e3f0a557-gw-openshift-default-7d94d6b5f7-fts5d\" (UID: \"3b980620-e583-42e5-b141-264c2b72bfb2\") " pod="openshift-ingress/e2e-migrate-e3f0a557-gw-openshift-default-7d94d6b5f7-fts5d" Nov 03 01:29:19.789125 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:19.789116 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b980620-e583-42e5-b141-264c2b72bfb2-istiod-ca-cert\") pod \"e2e-migrate-e3f0a557-gw-openshift-default-7d94d6b5f7-fts5d\" (UID: \"3b980620-e583-42e5-b141-264c2b72bfb2\") " pod="openshift-ingress/e2e-migrate-e3f0a557-gw-openshift-default-7d94d6b5f7-fts5d" Nov 03 01:29:19.790893 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:19.790873 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b980620-e583-42e5-b141-264c2b72bfb2-istio-envoy\") pod \"e2e-migrate-e3f0a557-gw-openshift-default-7d94d6b5f7-fts5d\" (UID: \"3b980620-e583-42e5-b141-264c2b72bfb2\") " pod="openshift-ingress/e2e-migrate-e3f0a557-gw-openshift-default-7d94d6b5f7-fts5d" Nov 03 01:29:19.791232 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:19.791205 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b980620-e583-42e5-b141-264c2b72bfb2-istio-podinfo\") pod \"e2e-migrate-e3f0a557-gw-openshift-default-7d94d6b5f7-fts5d\" (UID: \"3b980620-e583-42e5-b141-264c2b72bfb2\") " pod="openshift-ingress/e2e-migrate-e3f0a557-gw-openshift-default-7d94d6b5f7-fts5d" Nov 03 01:29:19.858055 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:19.858017 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b980620-e583-42e5-b141-264c2b72bfb2-istio-token\") pod \"e2e-migrate-e3f0a557-gw-openshift-default-7d94d6b5f7-fts5d\" (UID: \"3b980620-e583-42e5-b141-264c2b72bfb2\") " pod="openshift-ingress/e2e-migrate-e3f0a557-gw-openshift-default-7d94d6b5f7-fts5d" Nov 03 01:29:19.858209 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:19.858085 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48bmb\" (UniqueName: \"kubernetes.io/projected/3b980620-e583-42e5-b141-264c2b72bfb2-kube-api-access-48bmb\") pod \"e2e-migrate-e3f0a557-gw-openshift-default-7d94d6b5f7-fts5d\" (UID: \"3b980620-e583-42e5-b141-264c2b72bfb2\") " pod="openshift-ingress/e2e-migrate-e3f0a557-gw-openshift-default-7d94d6b5f7-fts5d" Nov 03 01:29:19.875837 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:19.875797 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:20.094826 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:20.094781 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:20.141279 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:20.141237 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-e3f0a557-gw-openshift-default-7d94d6b5f7-fts5d" Nov 03 01:29:20.561596 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:29:20.561556 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b980620_e583_42e5_b141_264c2b72bfb2.slice/crio-6dfb85547d6f1397330ce2aaa95c9db74a445e15650693c6724ba06e129a211c WatchSource:0}: Error finding container 6dfb85547d6f1397330ce2aaa95c9db74a445e15650693c6724ba06e129a211c: Status 404 returned error can't find the container with id 6dfb85547d6f1397330ce2aaa95c9db74a445e15650693c6724ba06e129a211c Nov 03 01:29:20.563959 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:20.563933 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-e3f0a557-gw-openshift-default-7d94d6b5f7-fts5d"] Nov 03 01:29:20.564148 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:20.564117 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 01:29:20.564200 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:20.564187 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 01:29:20.564238 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:20.564220 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 01:29:20.859100 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:20.859013 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e3f0a557-gw-openshift-default-7d94d6b5f7-fts5d" event={"ID":"3b980620-e583-42e5-b141-264c2b72bfb2","Type":"ContainerStarted","Data":"bf6c89f4ee5d24971c8e12073f6781f0519e64fc9893b3464598824acda2df83"} Nov 03 01:29:20.859100 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:20.859064 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e3f0a557-gw-openshift-default-7d94d6b5f7-fts5d" event={"ID":"3b980620-e583-42e5-b141-264c2b72bfb2","Type":"ContainerStarted","Data":"6dfb85547d6f1397330ce2aaa95c9db74a445e15650693c6724ba06e129a211c"} Nov 03 01:29:20.982399 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:20.982328 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-e3f0a557-gw-openshift-default-7d94d6b5f7-fts5d" podStartSLOduration=1.982314792 podStartE2EDuration="1.982314792s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:29:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:29:20.98208183 +0000 UTC m=+643.011564126" watchObservedRunningTime="2026-11-03 01:29:20.982314792 +0000 UTC m=+643.011797083" Nov 03 01:29:21.142434 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:21.142308 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-e3f0a557-gw-openshift-default-7d94d6b5f7-fts5d" Nov 03 01:29:21.147657 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:21.147628 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-e3f0a557-gw-openshift-default-7d94d6b5f7-fts5d" Nov 03 01:29:21.420066 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:21.419970 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:21.669739 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:21.669698 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:21.864534 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:21.864498 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-e3f0a557-gw-openshift-default-7d94d6b5f7-fts5d" Nov 03 01:29:21.865496 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:21.865475 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-e3f0a557-gw-openshift-default-7d94d6b5f7-fts5d" Nov 03 01:29:23.006604 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:23.006558 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:23.271759 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:23.271671 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:23.660104 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:23.660056 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:23.877017 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:23.876979 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:24.646666 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:24.646627 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:24.900003 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:24.899885 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:25.533809 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:25.533768 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:25.738569 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:25.738533 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:25.945327 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:25.945285 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:26.319000 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:26.318959 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-1e406323-openshift-default-5fd6659f69-rs9dq"] Nov 03 01:29:26.322715 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:26.322690 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-1e406323-openshift-default-5fd6659f69-rs9dq" Nov 03 01:29:26.354079 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:26.354031 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/576f7646-8212-4ed1-b4f1-2d000cc4072b-workload-certs\") pod \"e2e-derive-1e406323-openshift-default-5fd6659f69-rs9dq\" (UID: \"576f7646-8212-4ed1-b4f1-2d000cc4072b\") " pod="openshift-ingress/e2e-derive-1e406323-openshift-default-5fd6659f69-rs9dq" Nov 03 01:29:26.354079 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:26.354070 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/576f7646-8212-4ed1-b4f1-2d000cc4072b-istio-data\") pod \"e2e-derive-1e406323-openshift-default-5fd6659f69-rs9dq\" (UID: \"576f7646-8212-4ed1-b4f1-2d000cc4072b\") " pod="openshift-ingress/e2e-derive-1e406323-openshift-default-5fd6659f69-rs9dq" Nov 03 01:29:26.354337 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:26.354105 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/576f7646-8212-4ed1-b4f1-2d000cc4072b-istio-token\") pod \"e2e-derive-1e406323-openshift-default-5fd6659f69-rs9dq\" (UID: \"576f7646-8212-4ed1-b4f1-2d000cc4072b\") " pod="openshift-ingress/e2e-derive-1e406323-openshift-default-5fd6659f69-rs9dq" Nov 03 01:29:26.354337 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:26.354219 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l44w6\" (UniqueName: \"kubernetes.io/projected/576f7646-8212-4ed1-b4f1-2d000cc4072b-kube-api-access-l44w6\") pod \"e2e-derive-1e406323-openshift-default-5fd6659f69-rs9dq\" (UID: \"576f7646-8212-4ed1-b4f1-2d000cc4072b\") " pod="openshift-ingress/e2e-derive-1e406323-openshift-default-5fd6659f69-rs9dq" Nov 03 01:29:26.354337 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:26.354262 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/576f7646-8212-4ed1-b4f1-2d000cc4072b-workload-socket\") pod \"e2e-derive-1e406323-openshift-default-5fd6659f69-rs9dq\" (UID: \"576f7646-8212-4ed1-b4f1-2d000cc4072b\") " pod="openshift-ingress/e2e-derive-1e406323-openshift-default-5fd6659f69-rs9dq" Nov 03 01:29:26.354552 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:26.354359 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/576f7646-8212-4ed1-b4f1-2d000cc4072b-istiod-ca-cert\") pod \"e2e-derive-1e406323-openshift-default-5fd6659f69-rs9dq\" (UID: \"576f7646-8212-4ed1-b4f1-2d000cc4072b\") " pod="openshift-ingress/e2e-derive-1e406323-openshift-default-5fd6659f69-rs9dq" Nov 03 01:29:26.354552 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:26.354472 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/576f7646-8212-4ed1-b4f1-2d000cc4072b-istio-envoy\") pod \"e2e-derive-1e406323-openshift-default-5fd6659f69-rs9dq\" (UID: \"576f7646-8212-4ed1-b4f1-2d000cc4072b\") " pod="openshift-ingress/e2e-derive-1e406323-openshift-default-5fd6659f69-rs9dq" Nov 03 01:29:26.354552 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:26.354541 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/576f7646-8212-4ed1-b4f1-2d000cc4072b-credential-socket\") pod \"e2e-derive-1e406323-openshift-default-5fd6659f69-rs9dq\" (UID: \"576f7646-8212-4ed1-b4f1-2d000cc4072b\") " pod="openshift-ingress/e2e-derive-1e406323-openshift-default-5fd6659f69-rs9dq" Nov 03 01:29:26.354709 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:26.354604 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/576f7646-8212-4ed1-b4f1-2d000cc4072b-istio-podinfo\") pod \"e2e-derive-1e406323-openshift-default-5fd6659f69-rs9dq\" (UID: \"576f7646-8212-4ed1-b4f1-2d000cc4072b\") " pod="openshift-ingress/e2e-derive-1e406323-openshift-default-5fd6659f69-rs9dq" Nov 03 01:29:26.378438 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:26.378407 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-1e406323-openshift-default-dockercfg-ncpll\"" Nov 03 01:29:26.431681 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:26.431646 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-1e406323-openshift-default-5fd6659f69-rs9dq"] Nov 03 01:29:26.455580 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:26.455535 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/576f7646-8212-4ed1-b4f1-2d000cc4072b-istio-podinfo\") pod \"e2e-derive-1e406323-openshift-default-5fd6659f69-rs9dq\" (UID: \"576f7646-8212-4ed1-b4f1-2d000cc4072b\") " pod="openshift-ingress/e2e-derive-1e406323-openshift-default-5fd6659f69-rs9dq" Nov 03 01:29:26.455778 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:26.455618 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/576f7646-8212-4ed1-b4f1-2d000cc4072b-workload-certs\") pod \"e2e-derive-1e406323-openshift-default-5fd6659f69-rs9dq\" (UID: \"576f7646-8212-4ed1-b4f1-2d000cc4072b\") " pod="openshift-ingress/e2e-derive-1e406323-openshift-default-5fd6659f69-rs9dq" Nov 03 01:29:26.455778 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:26.455650 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/576f7646-8212-4ed1-b4f1-2d000cc4072b-istio-data\") pod \"e2e-derive-1e406323-openshift-default-5fd6659f69-rs9dq\" (UID: \"576f7646-8212-4ed1-b4f1-2d000cc4072b\") " pod="openshift-ingress/e2e-derive-1e406323-openshift-default-5fd6659f69-rs9dq" Nov 03 01:29:26.455778 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:26.455693 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/576f7646-8212-4ed1-b4f1-2d000cc4072b-istio-token\") pod \"e2e-derive-1e406323-openshift-default-5fd6659f69-rs9dq\" (UID: \"576f7646-8212-4ed1-b4f1-2d000cc4072b\") " pod="openshift-ingress/e2e-derive-1e406323-openshift-default-5fd6659f69-rs9dq" Nov 03 01:29:26.455778 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:26.455730 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l44w6\" (UniqueName: \"kubernetes.io/projected/576f7646-8212-4ed1-b4f1-2d000cc4072b-kube-api-access-l44w6\") pod \"e2e-derive-1e406323-openshift-default-5fd6659f69-rs9dq\" (UID: \"576f7646-8212-4ed1-b4f1-2d000cc4072b\") " pod="openshift-ingress/e2e-derive-1e406323-openshift-default-5fd6659f69-rs9dq" Nov 03 01:29:26.455778 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:26.455760 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/576f7646-8212-4ed1-b4f1-2d000cc4072b-workload-socket\") pod \"e2e-derive-1e406323-openshift-default-5fd6659f69-rs9dq\" (UID: \"576f7646-8212-4ed1-b4f1-2d000cc4072b\") " pod="openshift-ingress/e2e-derive-1e406323-openshift-default-5fd6659f69-rs9dq" Nov 03 01:29:26.455960 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:26.455796 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/576f7646-8212-4ed1-b4f1-2d000cc4072b-istiod-ca-cert\") pod \"e2e-derive-1e406323-openshift-default-5fd6659f69-rs9dq\" (UID: \"576f7646-8212-4ed1-b4f1-2d000cc4072b\") " pod="openshift-ingress/e2e-derive-1e406323-openshift-default-5fd6659f69-rs9dq" Nov 03 01:29:26.455960 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:26.455864 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/576f7646-8212-4ed1-b4f1-2d000cc4072b-istio-envoy\") pod \"e2e-derive-1e406323-openshift-default-5fd6659f69-rs9dq\" (UID: \"576f7646-8212-4ed1-b4f1-2d000cc4072b\") " pod="openshift-ingress/e2e-derive-1e406323-openshift-default-5fd6659f69-rs9dq" Nov 03 01:29:26.455960 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:26.455906 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/576f7646-8212-4ed1-b4f1-2d000cc4072b-credential-socket\") pod \"e2e-derive-1e406323-openshift-default-5fd6659f69-rs9dq\" (UID: \"576f7646-8212-4ed1-b4f1-2d000cc4072b\") " pod="openshift-ingress/e2e-derive-1e406323-openshift-default-5fd6659f69-rs9dq" Nov 03 01:29:26.456417 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:26.456384 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/576f7646-8212-4ed1-b4f1-2d000cc4072b-credential-socket\") pod \"e2e-derive-1e406323-openshift-default-5fd6659f69-rs9dq\" (UID: \"576f7646-8212-4ed1-b4f1-2d000cc4072b\") " pod="openshift-ingress/e2e-derive-1e406323-openshift-default-5fd6659f69-rs9dq" Nov 03 01:29:26.456927 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:26.456869 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/576f7646-8212-4ed1-b4f1-2d000cc4072b-istio-data\") pod \"e2e-derive-1e406323-openshift-default-5fd6659f69-rs9dq\" (UID: \"576f7646-8212-4ed1-b4f1-2d000cc4072b\") " pod="openshift-ingress/e2e-derive-1e406323-openshift-default-5fd6659f69-rs9dq" Nov 03 01:29:26.457479 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:26.457393 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/576f7646-8212-4ed1-b4f1-2d000cc4072b-workload-socket\") pod \"e2e-derive-1e406323-openshift-default-5fd6659f69-rs9dq\" (UID: \"576f7646-8212-4ed1-b4f1-2d000cc4072b\") " pod="openshift-ingress/e2e-derive-1e406323-openshift-default-5fd6659f69-rs9dq" Nov 03 01:29:26.457479 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:26.457422 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/576f7646-8212-4ed1-b4f1-2d000cc4072b-workload-certs\") pod \"e2e-derive-1e406323-openshift-default-5fd6659f69-rs9dq\" (UID: \"576f7646-8212-4ed1-b4f1-2d000cc4072b\") " pod="openshift-ingress/e2e-derive-1e406323-openshift-default-5fd6659f69-rs9dq" Nov 03 01:29:26.457779 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:26.457759 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/576f7646-8212-4ed1-b4f1-2d000cc4072b-istiod-ca-cert\") pod \"e2e-derive-1e406323-openshift-default-5fd6659f69-rs9dq\" (UID: \"576f7646-8212-4ed1-b4f1-2d000cc4072b\") " pod="openshift-ingress/e2e-derive-1e406323-openshift-default-5fd6659f69-rs9dq" Nov 03 01:29:26.459639 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:26.459605 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/576f7646-8212-4ed1-b4f1-2d000cc4072b-istio-podinfo\") pod \"e2e-derive-1e406323-openshift-default-5fd6659f69-rs9dq\" (UID: \"576f7646-8212-4ed1-b4f1-2d000cc4072b\") " pod="openshift-ingress/e2e-derive-1e406323-openshift-default-5fd6659f69-rs9dq" Nov 03 01:29:26.459741 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:26.459612 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/576f7646-8212-4ed1-b4f1-2d000cc4072b-istio-envoy\") pod \"e2e-derive-1e406323-openshift-default-5fd6659f69-rs9dq\" (UID: \"576f7646-8212-4ed1-b4f1-2d000cc4072b\") " pod="openshift-ingress/e2e-derive-1e406323-openshift-default-5fd6659f69-rs9dq" Nov 03 01:29:26.460870 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:26.460842 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:26.519600 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:26.519557 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/576f7646-8212-4ed1-b4f1-2d000cc4072b-istio-token\") pod \"e2e-derive-1e406323-openshift-default-5fd6659f69-rs9dq\" (UID: \"576f7646-8212-4ed1-b4f1-2d000cc4072b\") " pod="openshift-ingress/e2e-derive-1e406323-openshift-default-5fd6659f69-rs9dq" Nov 03 01:29:26.519867 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:26.519843 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l44w6\" (UniqueName: \"kubernetes.io/projected/576f7646-8212-4ed1-b4f1-2d000cc4072b-kube-api-access-l44w6\") pod \"e2e-derive-1e406323-openshift-default-5fd6659f69-rs9dq\" (UID: \"576f7646-8212-4ed1-b4f1-2d000cc4072b\") " pod="openshift-ingress/e2e-derive-1e406323-openshift-default-5fd6659f69-rs9dq" Nov 03 01:29:26.636109 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:26.636014 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-1e406323-openshift-default-5fd6659f69-rs9dq" Nov 03 01:29:26.668080 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:26.668038 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:26.985564 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:26.985463 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:27.056873 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:27.056836 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-1e406323-openshift-default-5fd6659f69-rs9dq"] Nov 03 01:29:27.087200 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:29:27.087170 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod576f7646_8212_4ed1_b4f1_2d000cc4072b.slice/crio-b889b0cf5d7d18519630d41d9b6befec7bd9798e04c06e35a4a5798afa06195a WatchSource:0}: Error finding container b889b0cf5d7d18519630d41d9b6befec7bd9798e04c06e35a4a5798afa06195a: Status 404 returned error can't find the container with id b889b0cf5d7d18519630d41d9b6befec7bd9798e04c06e35a4a5798afa06195a Nov 03 01:29:27.089587 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:27.089553 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 01:29:27.089667 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:27.089625 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 01:29:27.089667 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:27.089653 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 01:29:27.187826 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:27.187792 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:27.478805 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:27.478768 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-e3f0a557-565f8678bf-lgs4j"] Nov 03 01:29:27.482438 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:27.482407 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-e3f0a557-565f8678bf-lgs4j" Nov 03 01:29:27.536802 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:27.536753 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-e3f0a557-565f8678bf-lgs4j"] Nov 03 01:29:27.537357 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:27.537333 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-e3f0a557-dockercfg-8hc4x\"" Nov 03 01:29:27.537513 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:27.537480 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-e3f0a557\"" Nov 03 01:29:27.566974 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:27.566941 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd7a749c-b42c-4deb-904b-0bbe6d34ee70-plugins-config-volume\") pod \"payload-processing-e2e-migrate-e3f0a557-565f8678bf-lgs4j\" (UID: \"fd7a749c-b42c-4deb-904b-0bbe6d34ee70\") " pod="openshift-ingress/payload-processing-e2e-migrate-e3f0a557-565f8678bf-lgs4j" Nov 03 01:29:27.567154 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:27.567049 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8pf8\" (UniqueName: \"kubernetes.io/projected/fd7a749c-b42c-4deb-904b-0bbe6d34ee70-kube-api-access-n8pf8\") pod \"payload-processing-e2e-migrate-e3f0a557-565f8678bf-lgs4j\" (UID: \"fd7a749c-b42c-4deb-904b-0bbe6d34ee70\") " pod="openshift-ingress/payload-processing-e2e-migrate-e3f0a557-565f8678bf-lgs4j" Nov 03 01:29:27.668095 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:27.668057 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8pf8\" (UniqueName: \"kubernetes.io/projected/fd7a749c-b42c-4deb-904b-0bbe6d34ee70-kube-api-access-n8pf8\") pod \"payload-processing-e2e-migrate-e3f0a557-565f8678bf-lgs4j\" (UID: \"fd7a749c-b42c-4deb-904b-0bbe6d34ee70\") " pod="openshift-ingress/payload-processing-e2e-migrate-e3f0a557-565f8678bf-lgs4j" Nov 03 01:29:27.668256 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:27.668148 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd7a749c-b42c-4deb-904b-0bbe6d34ee70-plugins-config-volume\") pod \"payload-processing-e2e-migrate-e3f0a557-565f8678bf-lgs4j\" (UID: \"fd7a749c-b42c-4deb-904b-0bbe6d34ee70\") " pod="openshift-ingress/payload-processing-e2e-migrate-e3f0a557-565f8678bf-lgs4j" Nov 03 01:29:27.668870 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:27.668850 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd7a749c-b42c-4deb-904b-0bbe6d34ee70-plugins-config-volume\") pod \"payload-processing-e2e-migrate-e3f0a557-565f8678bf-lgs4j\" (UID: \"fd7a749c-b42c-4deb-904b-0bbe6d34ee70\") " pod="openshift-ingress/payload-processing-e2e-migrate-e3f0a557-565f8678bf-lgs4j" Nov 03 01:29:27.731060 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:27.730984 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8pf8\" (UniqueName: \"kubernetes.io/projected/fd7a749c-b42c-4deb-904b-0bbe6d34ee70-kube-api-access-n8pf8\") pod \"payload-processing-e2e-migrate-e3f0a557-565f8678bf-lgs4j\" (UID: \"fd7a749c-b42c-4deb-904b-0bbe6d34ee70\") " pod="openshift-ingress/payload-processing-e2e-migrate-e3f0a557-565f8678bf-lgs4j" Nov 03 01:29:27.798204 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:27.798161 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-e3f0a557-565f8678bf-lgs4j" Nov 03 01:29:27.897005 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:27.896960 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-1e406323-openshift-default-5fd6659f69-rs9dq" event={"ID":"576f7646-8212-4ed1-b4f1-2d000cc4072b","Type":"ContainerStarted","Data":"07efcd1440f2d9ab1a3fd5dfa898f5f3fb995fdce8e0091bc90a4dcce598b4e0"} Nov 03 01:29:27.897201 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:27.897051 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-1e406323-openshift-default-5fd6659f69-rs9dq" event={"ID":"576f7646-8212-4ed1-b4f1-2d000cc4072b","Type":"ContainerStarted","Data":"b889b0cf5d7d18519630d41d9b6befec7bd9798e04c06e35a4a5798afa06195a"} Nov 03 01:29:28.022950 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:28.022831 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-1e406323-openshift-default-5fd6659f69-rs9dq" podStartSLOduration=2.022817631 podStartE2EDuration="2.022817631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:29:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:29:28.02203308 +0000 UTC m=+650.051515374" watchObservedRunningTime="2026-11-03 01:29:28.022817631 +0000 UTC m=+650.052299925" Nov 03 01:29:28.229675 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:29:28.229638 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd7a749c_b42c_4deb_904b_0bbe6d34ee70.slice/crio-90e7aef21efffc9fb9cc1f0915aefa26ad8001adf6daee4190e1c10f7b0de43f WatchSource:0}: Error finding container 90e7aef21efffc9fb9cc1f0915aefa26ad8001adf6daee4190e1c10f7b0de43f: Status 404 returned error can't find the container with id 90e7aef21efffc9fb9cc1f0915aefa26ad8001adf6daee4190e1c10f7b0de43f Nov 03 01:29:28.231438 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:28.231409 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-e3f0a557-565f8678bf-lgs4j"] Nov 03 01:29:28.636646 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:28.636594 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-1e406323-openshift-default-5fd6659f69-rs9dq" Nov 03 01:29:28.642027 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:28.641995 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-1e406323-openshift-default-5fd6659f69-rs9dq" Nov 03 01:29:28.903893 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:28.903787 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-e3f0a557-565f8678bf-lgs4j" event={"ID":"fd7a749c-b42c-4deb-904b-0bbe6d34ee70","Type":"ContainerStarted","Data":"32bab3f843e4d6da88b7f7d9be093547b1f22c3092fd1e38823067e7b0f9d775"} Nov 03 01:29:28.903893 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:28.903852 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-e3f0a557-565f8678bf-lgs4j" event={"ID":"fd7a749c-b42c-4deb-904b-0bbe6d34ee70","Type":"ContainerStarted","Data":"90e7aef21efffc9fb9cc1f0915aefa26ad8001adf6daee4190e1c10f7b0de43f"} Nov 03 01:29:28.904325 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:28.904031 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-1e406323-openshift-default-5fd6659f69-rs9dq" Nov 03 01:29:28.904325 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:28.904081 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-e3f0a557-565f8678bf-lgs4j" Nov 03 01:29:28.905112 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:28.905093 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-1e406323-openshift-default-5fd6659f69-rs9dq" Nov 03 01:29:28.916297 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:28.916264 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:29.025117 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:29.025058 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-e3f0a557-565f8678bf-lgs4j" podStartSLOduration=2.025044421 podStartE2EDuration="2.025044421s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:29:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:29:29.024324645 +0000 UTC m=+651.053807004" watchObservedRunningTime="2026-11-03 01:29:29.025044421 +0000 UTC m=+651.054526716" Nov 03 01:29:29.131049 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:29.131007 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:29.430777 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:29.430732 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:29.645213 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:29.645176 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:29.851898 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:29.851862 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:32.144147 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:32.144111 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-21c25193-64474f487f-mndts"] Nov 03 01:29:32.148181 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:32.148156 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-21c25193-64474f487f-mndts" Nov 03 01:29:32.204218 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:32.204184 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-21c25193-64474f487f-mndts"] Nov 03 01:29:32.212726 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:32.212691 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccfd2cf0-8560-48c6-b075-2d5f65f81f5c-plugins-config-volume\") pod \"payload-processing-e2e-ait-21c25193-64474f487f-mndts\" (UID: \"ccfd2cf0-8560-48c6-b075-2d5f65f81f5c\") " pod="openshift-ingress/payload-processing-e2e-ait-21c25193-64474f487f-mndts" Nov 03 01:29:32.212891 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:32.212801 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdtwv\" (UniqueName: \"kubernetes.io/projected/ccfd2cf0-8560-48c6-b075-2d5f65f81f5c-kube-api-access-tdtwv\") pod \"payload-processing-e2e-ait-21c25193-64474f487f-mndts\" (UID: \"ccfd2cf0-8560-48c6-b075-2d5f65f81f5c\") " pod="openshift-ingress/payload-processing-e2e-ait-21c25193-64474f487f-mndts" Nov 03 01:29:32.253555 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:32.253521 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-21c25193\"" Nov 03 01:29:32.253720 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:32.253589 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-21c25193-dockercfg-hmc5h\"" Nov 03 01:29:32.313719 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:32.313640 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdtwv\" (UniqueName: \"kubernetes.io/projected/ccfd2cf0-8560-48c6-b075-2d5f65f81f5c-kube-api-access-tdtwv\") pod \"payload-processing-e2e-ait-21c25193-64474f487f-mndts\" (UID: \"ccfd2cf0-8560-48c6-b075-2d5f65f81f5c\") " pod="openshift-ingress/payload-processing-e2e-ait-21c25193-64474f487f-mndts" Nov 03 01:29:32.313954 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:32.313881 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccfd2cf0-8560-48c6-b075-2d5f65f81f5c-plugins-config-volume\") pod \"payload-processing-e2e-ait-21c25193-64474f487f-mndts\" (UID: \"ccfd2cf0-8560-48c6-b075-2d5f65f81f5c\") " pod="openshift-ingress/payload-processing-e2e-ait-21c25193-64474f487f-mndts" Nov 03 01:29:32.314790 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:32.314766 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccfd2cf0-8560-48c6-b075-2d5f65f81f5c-plugins-config-volume\") pod \"payload-processing-e2e-ait-21c25193-64474f487f-mndts\" (UID: \"ccfd2cf0-8560-48c6-b075-2d5f65f81f5c\") " pod="openshift-ingress/payload-processing-e2e-ait-21c25193-64474f487f-mndts" Nov 03 01:29:32.376719 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:32.376691 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdtwv\" (UniqueName: \"kubernetes.io/projected/ccfd2cf0-8560-48c6-b075-2d5f65f81f5c-kube-api-access-tdtwv\") pod \"payload-processing-e2e-ait-21c25193-64474f487f-mndts\" (UID: \"ccfd2cf0-8560-48c6-b075-2d5f65f81f5c\") " pod="openshift-ingress/payload-processing-e2e-ait-21c25193-64474f487f-mndts" Nov 03 01:29:32.462187 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:32.462074 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-21c25193-64474f487f-mndts" Nov 03 01:29:32.665526 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:32.665472 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:32.837508 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:29:32.837469 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccfd2cf0_8560_48c6_b075_2d5f65f81f5c.slice/crio-4732a7fba71a4b98859e83e8cd361d8a1c860d8cabaf8ffc81c37012f9158e27 WatchSource:0}: Error finding container 4732a7fba71a4b98859e83e8cd361d8a1c860d8cabaf8ffc81c37012f9158e27: Status 404 returned error can't find the container with id 4732a7fba71a4b98859e83e8cd361d8a1c860d8cabaf8ffc81c37012f9158e27 Nov 03 01:29:32.840143 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:32.840116 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-21c25193-64474f487f-mndts"] Nov 03 01:29:32.927448 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:32.927410 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-21c25193-64474f487f-mndts" event={"ID":"ccfd2cf0-8560-48c6-b075-2d5f65f81f5c","Type":"ContainerStarted","Data":"4b051987972220988c216e7b032072c3fdec2efcb67ef54c09e4ab064df8ff7e"} Nov 03 01:29:32.927635 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:32.927472 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-21c25193-64474f487f-mndts" event={"ID":"ccfd2cf0-8560-48c6-b075-2d5f65f81f5c","Type":"ContainerStarted","Data":"4732a7fba71a4b98859e83e8cd361d8a1c860d8cabaf8ffc81c37012f9158e27"} Nov 03 01:29:32.927635 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:32.927627 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-21c25193-64474f487f-mndts" Nov 03 01:29:32.934403 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:32.934332 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:33.050895 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:33.050834 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-21c25193-64474f487f-mndts" podStartSLOduration=1.05081849 podStartE2EDuration="1.05081849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:29:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:29:33.048869041 +0000 UTC m=+655.078351336" watchObservedRunningTime="2026-11-03 01:29:33.05081849 +0000 UTC m=+655.080300823" Nov 03 01:29:33.143389 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:33.143262 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:33.992455 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:33.992418 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:34.249112 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:34.249001 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:34.345431 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:34.345392 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-98c7cd16-5799c4dc6c-nz2xm"] Nov 03 01:29:34.349181 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:34.349147 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-98c7cd16-5799c4dc6c-nz2xm" Nov 03 01:29:34.418080 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:34.418043 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-98c7cd16-5799c4dc6c-nz2xm"] Nov 03 01:29:34.437088 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:34.437043 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9946518f-161b-4ef2-b852-e7be15ed73de-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-98c7cd16-5799c4dc6c-nz2xm\" (UID: \"9946518f-161b-4ef2-b852-e7be15ed73de\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-98c7cd16-5799c4dc6c-nz2xm" Nov 03 01:29:34.437275 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:34.437209 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzthd\" (UniqueName: \"kubernetes.io/projected/9946518f-161b-4ef2-b852-e7be15ed73de-kube-api-access-hzthd\") pod \"payload-pre-processing-e2e-ait-98c7cd16-5799c4dc6c-nz2xm\" (UID: \"9946518f-161b-4ef2-b852-e7be15ed73de\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-98c7cd16-5799c4dc6c-nz2xm" Nov 03 01:29:34.465949 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:34.465906 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-98c7cd16-dockercfg-krgrs\"" Nov 03 01:29:34.465949 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:34.465914 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-98c7cd16\"" Nov 03 01:29:34.503269 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:34.503176 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-98c7cd16-d886f8d45-t5dzj"] Nov 03 01:29:34.507240 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:34.507216 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-98c7cd16-d886f8d45-t5dzj" Nov 03 01:29:34.538755 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:34.538709 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9946518f-161b-4ef2-b852-e7be15ed73de-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-98c7cd16-5799c4dc6c-nz2xm\" (UID: \"9946518f-161b-4ef2-b852-e7be15ed73de\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-98c7cd16-5799c4dc6c-nz2xm" Nov 03 01:29:34.538958 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:34.538768 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b6e29e8-e823-4a46-82cf-8b3f7720b934-plugins-config-volume\") pod \"payload-processing-e2e-ait-98c7cd16-d886f8d45-t5dzj\" (UID: \"2b6e29e8-e823-4a46-82cf-8b3f7720b934\") " pod="openshift-ingress/payload-processing-e2e-ait-98c7cd16-d886f8d45-t5dzj" Nov 03 01:29:34.538958 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:34.538818 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2gmm\" (UniqueName: \"kubernetes.io/projected/2b6e29e8-e823-4a46-82cf-8b3f7720b934-kube-api-access-d2gmm\") pod \"payload-processing-e2e-ait-98c7cd16-d886f8d45-t5dzj\" (UID: \"2b6e29e8-e823-4a46-82cf-8b3f7720b934\") " pod="openshift-ingress/payload-processing-e2e-ait-98c7cd16-d886f8d45-t5dzj" Nov 03 01:29:34.538958 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:34.538859 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzthd\" (UniqueName: \"kubernetes.io/projected/9946518f-161b-4ef2-b852-e7be15ed73de-kube-api-access-hzthd\") pod \"payload-pre-processing-e2e-ait-98c7cd16-5799c4dc6c-nz2xm\" (UID: \"9946518f-161b-4ef2-b852-e7be15ed73de\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-98c7cd16-5799c4dc6c-nz2xm" Nov 03 01:29:34.539603 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:34.539582 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9946518f-161b-4ef2-b852-e7be15ed73de-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-98c7cd16-5799c4dc6c-nz2xm\" (UID: \"9946518f-161b-4ef2-b852-e7be15ed73de\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-98c7cd16-5799c4dc6c-nz2xm" Nov 03 01:29:34.561575 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:34.561535 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-98c7cd16-d886f8d45-t5dzj"] Nov 03 01:29:34.611157 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:34.611114 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzthd\" (UniqueName: \"kubernetes.io/projected/9946518f-161b-4ef2-b852-e7be15ed73de-kube-api-access-hzthd\") pod \"payload-pre-processing-e2e-ait-98c7cd16-5799c4dc6c-nz2xm\" (UID: \"9946518f-161b-4ef2-b852-e7be15ed73de\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-98c7cd16-5799c4dc6c-nz2xm" Nov 03 01:29:34.639765 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:34.639728 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b6e29e8-e823-4a46-82cf-8b3f7720b934-plugins-config-volume\") pod \"payload-processing-e2e-ait-98c7cd16-d886f8d45-t5dzj\" (UID: \"2b6e29e8-e823-4a46-82cf-8b3f7720b934\") " pod="openshift-ingress/payload-processing-e2e-ait-98c7cd16-d886f8d45-t5dzj" Nov 03 01:29:34.639948 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:34.639786 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2gmm\" (UniqueName: \"kubernetes.io/projected/2b6e29e8-e823-4a46-82cf-8b3f7720b934-kube-api-access-d2gmm\") pod \"payload-processing-e2e-ait-98c7cd16-d886f8d45-t5dzj\" (UID: \"2b6e29e8-e823-4a46-82cf-8b3f7720b934\") " pod="openshift-ingress/payload-processing-e2e-ait-98c7cd16-d886f8d45-t5dzj" Nov 03 01:29:34.640558 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:34.640535 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b6e29e8-e823-4a46-82cf-8b3f7720b934-plugins-config-volume\") pod \"payload-processing-e2e-ait-98c7cd16-d886f8d45-t5dzj\" (UID: \"2b6e29e8-e823-4a46-82cf-8b3f7720b934\") " pod="openshift-ingress/payload-processing-e2e-ait-98c7cd16-d886f8d45-t5dzj" Nov 03 01:29:34.663507 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:34.663379 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-98c7cd16-5799c4dc6c-nz2xm" Nov 03 01:29:34.710724 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:34.710662 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:34.713726 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:34.713701 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2gmm\" (UniqueName: \"kubernetes.io/projected/2b6e29e8-e823-4a46-82cf-8b3f7720b934-kube-api-access-d2gmm\") pod \"payload-processing-e2e-ait-98c7cd16-d886f8d45-t5dzj\" (UID: \"2b6e29e8-e823-4a46-82cf-8b3f7720b934\") " pod="openshift-ingress/payload-processing-e2e-ait-98c7cd16-d886f8d45-t5dzj" Nov 03 01:29:34.821309 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:34.821269 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-98c7cd16-d886f8d45-t5dzj" Nov 03 01:29:34.911117 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:34.911078 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:35.052272 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:29:35.052231 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9946518f_161b_4ef2_b852_e7be15ed73de.slice/crio-8fd6a70dd0511183153007f5e7c98a373b12bc08ecedb2318c7c351da2a61d6e WatchSource:0}: Error finding container 8fd6a70dd0511183153007f5e7c98a373b12bc08ecedb2318c7c351da2a61d6e: Status 404 returned error can't find the container with id 8fd6a70dd0511183153007f5e7c98a373b12bc08ecedb2318c7c351da2a61d6e Nov 03 01:29:35.054391 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:35.054343 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-98c7cd16-5799c4dc6c-nz2xm"] Nov 03 01:29:35.202488 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:29:35.202455 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b6e29e8_e823_4a46_82cf_8b3f7720b934.slice/crio-98f36588ef1cafb42ffa68569bb991ad7ba9b52c677ba1af2bbaf3bc5d6ea9db WatchSource:0}: Error finding container 98f36588ef1cafb42ffa68569bb991ad7ba9b52c677ba1af2bbaf3bc5d6ea9db: Status 404 returned error can't find the container with id 98f36588ef1cafb42ffa68569bb991ad7ba9b52c677ba1af2bbaf3bc5d6ea9db Nov 03 01:29:35.204766 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:35.204736 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-98c7cd16-d886f8d45-t5dzj"] Nov 03 01:29:35.947485 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:35.947431 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-98c7cd16-5799c4dc6c-nz2xm" event={"ID":"9946518f-161b-4ef2-b852-e7be15ed73de","Type":"ContainerStarted","Data":"6ed5b43a6780b2dc6d933ad57420c33772d5941bf45cc5b8fc245d8620e16974"} Nov 03 01:29:35.947485 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:35.947491 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-98c7cd16-5799c4dc6c-nz2xm" event={"ID":"9946518f-161b-4ef2-b852-e7be15ed73de","Type":"ContainerStarted","Data":"8fd6a70dd0511183153007f5e7c98a373b12bc08ecedb2318c7c351da2a61d6e"} Nov 03 01:29:35.947736 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:35.947515 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-98c7cd16-5799c4dc6c-nz2xm" Nov 03 01:29:35.948998 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:35.948976 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-98c7cd16-d886f8d45-t5dzj" event={"ID":"2b6e29e8-e823-4a46-82cf-8b3f7720b934","Type":"ContainerStarted","Data":"7c89c98f39aa826e8b69e9cf5e2fe60a9a445e0e13eaee71aca67dcf6bdcb39b"} Nov 03 01:29:35.949094 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:35.949013 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-98c7cd16-d886f8d45-t5dzj" event={"ID":"2b6e29e8-e823-4a46-82cf-8b3f7720b934","Type":"ContainerStarted","Data":"98f36588ef1cafb42ffa68569bb991ad7ba9b52c677ba1af2bbaf3bc5d6ea9db"} Nov 03 01:29:35.949094 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:35.949033 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-98c7cd16-d886f8d45-t5dzj" Nov 03 01:29:36.067813 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:36.067758 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-98c7cd16-5799c4dc6c-nz2xm" podStartSLOduration=2.067745087 podStartE2EDuration="2.067745087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:29:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:29:36.067161035 +0000 UTC m=+658.096643330" watchObservedRunningTime="2026-11-03 01:29:36.067745087 +0000 UTC m=+658.097227382" Nov 03 01:29:36.194261 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:36.194192 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-98c7cd16-d886f8d45-t5dzj" podStartSLOduration=2.194175037 podStartE2EDuration="2.194175037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:29:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:29:36.192355742 +0000 UTC m=+658.221838040" watchObservedRunningTime="2026-11-03 01:29:36.194175037 +0000 UTC m=+658.223657332" Nov 03 01:29:36.410660 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:36.410613 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:36.610232 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:36.610187 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:39.306113 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:39.306071 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:39.513666 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:39.513625 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:39.713452 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:39.713352 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:39.913247 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:39.913219 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-e3f0a557-565f8678bf-lgs4j" Nov 03 01:29:39.913581 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:39.913557 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:43.946697 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:43.946670 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-21c25193-64474f487f-mndts" Nov 03 01:29:44.056867 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:29:44.056829 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 01:29:44.528588 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:44.528549 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:45.543949 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:45.543628 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:45.750117 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:45.749971 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:45.880858 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:45.880821 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-1e406323-6cf7f67fc9-25xxb"] Nov 03 01:29:45.885942 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:45.885918 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-1e406323-6cf7f67fc9-25xxb" Nov 03 01:29:45.941393 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:45.941273 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-1e406323-6cf7f67fc9-25xxb"] Nov 03 01:29:45.983063 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:45.983015 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a932b015-5506-4653-89fb-4a1d9f1c002e-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-1e406323-6cf7f67fc9-25xxb\" (UID: \"a932b015-5506-4653-89fb-4a1d9f1c002e\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-1e406323-6cf7f67fc9-25xxb" Nov 03 01:29:45.983242 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:45.983078 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d99vp\" (UniqueName: \"kubernetes.io/projected/a932b015-5506-4653-89fb-4a1d9f1c002e-kube-api-access-d99vp\") pod \"payload-pre-processing-e2e-derive-1e406323-6cf7f67fc9-25xxb\" (UID: \"a932b015-5506-4653-89fb-4a1d9f1c002e\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-1e406323-6cf7f67fc9-25xxb" Nov 03 01:29:45.989678 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:45.989644 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-1e406323\"" Nov 03 01:29:45.989678 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:45.989659 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-1e406323-dockercfg-8c429\"" Nov 03 01:29:46.041492 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:46.041454 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-1e406323-597d58cc5f-qqrhb"] Nov 03 01:29:46.046447 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:46.046416 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-1e406323-597d58cc5f-qqrhb" Nov 03 01:29:46.083775 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:46.083739 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a932b015-5506-4653-89fb-4a1d9f1c002e-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-1e406323-6cf7f67fc9-25xxb\" (UID: \"a932b015-5506-4653-89fb-4a1d9f1c002e\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-1e406323-6cf7f67fc9-25xxb" Nov 03 01:29:46.083775 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:46.083786 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d99vp\" (UniqueName: \"kubernetes.io/projected/a932b015-5506-4653-89fb-4a1d9f1c002e-kube-api-access-d99vp\") pod \"payload-pre-processing-e2e-derive-1e406323-6cf7f67fc9-25xxb\" (UID: \"a932b015-5506-4653-89fb-4a1d9f1c002e\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-1e406323-6cf7f67fc9-25xxb" Nov 03 01:29:46.084787 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:46.084763 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a932b015-5506-4653-89fb-4a1d9f1c002e-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-1e406323-6cf7f67fc9-25xxb\" (UID: \"a932b015-5506-4653-89fb-4a1d9f1c002e\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-1e406323-6cf7f67fc9-25xxb" Nov 03 01:29:46.093160 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:46.093125 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-1e406323-597d58cc5f-qqrhb"] Nov 03 01:29:46.148036 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:46.147940 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d99vp\" (UniqueName: \"kubernetes.io/projected/a932b015-5506-4653-89fb-4a1d9f1c002e-kube-api-access-d99vp\") pod \"payload-pre-processing-e2e-derive-1e406323-6cf7f67fc9-25xxb\" (UID: \"a932b015-5506-4653-89fb-4a1d9f1c002e\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-1e406323-6cf7f67fc9-25xxb" Nov 03 01:29:46.184982 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:46.184945 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8hvd\" (UniqueName: \"kubernetes.io/projected/e9844242-b55f-4bc2-9861-50ea42f7bc90-kube-api-access-f8hvd\") pod \"payload-processing-e2e-derive-1e406323-597d58cc5f-qqrhb\" (UID: \"e9844242-b55f-4bc2-9861-50ea42f7bc90\") " pod="openshift-ingress/payload-processing-e2e-derive-1e406323-597d58cc5f-qqrhb" Nov 03 01:29:46.185175 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:46.185081 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9844242-b55f-4bc2-9861-50ea42f7bc90-plugins-config-volume\") pod \"payload-processing-e2e-derive-1e406323-597d58cc5f-qqrhb\" (UID: \"e9844242-b55f-4bc2-9861-50ea42f7bc90\") " pod="openshift-ingress/payload-processing-e2e-derive-1e406323-597d58cc5f-qqrhb" Nov 03 01:29:46.199882 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:46.199841 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-1e406323-6cf7f67fc9-25xxb" Nov 03 01:29:46.286184 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:46.286137 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8hvd\" (UniqueName: \"kubernetes.io/projected/e9844242-b55f-4bc2-9861-50ea42f7bc90-kube-api-access-f8hvd\") pod \"payload-processing-e2e-derive-1e406323-597d58cc5f-qqrhb\" (UID: \"e9844242-b55f-4bc2-9861-50ea42f7bc90\") " pod="openshift-ingress/payload-processing-e2e-derive-1e406323-597d58cc5f-qqrhb" Nov 03 01:29:46.286359 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:46.286309 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9844242-b55f-4bc2-9861-50ea42f7bc90-plugins-config-volume\") pod \"payload-processing-e2e-derive-1e406323-597d58cc5f-qqrhb\" (UID: \"e9844242-b55f-4bc2-9861-50ea42f7bc90\") " pod="openshift-ingress/payload-processing-e2e-derive-1e406323-597d58cc5f-qqrhb" Nov 03 01:29:46.287039 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:46.287008 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9844242-b55f-4bc2-9861-50ea42f7bc90-plugins-config-volume\") pod \"payload-processing-e2e-derive-1e406323-597d58cc5f-qqrhb\" (UID: \"e9844242-b55f-4bc2-9861-50ea42f7bc90\") " pod="openshift-ingress/payload-processing-e2e-derive-1e406323-597d58cc5f-qqrhb" Nov 03 01:29:46.349663 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:46.349627 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8hvd\" (UniqueName: \"kubernetes.io/projected/e9844242-b55f-4bc2-9861-50ea42f7bc90-kube-api-access-f8hvd\") pod \"payload-processing-e2e-derive-1e406323-597d58cc5f-qqrhb\" (UID: \"e9844242-b55f-4bc2-9861-50ea42f7bc90\") " pod="openshift-ingress/payload-processing-e2e-derive-1e406323-597d58cc5f-qqrhb" Nov 03 01:29:46.361946 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:46.361912 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-1e406323-597d58cc5f-qqrhb" Nov 03 01:29:46.587307 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:29:46.587275 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda932b015_5506_4653_89fb_4a1d9f1c002e.slice/crio-edc4be7eb81dd54b0744d918529a1ea6881482d8ecf3bc9855b534628bc2a7c1 WatchSource:0}: Error finding container edc4be7eb81dd54b0744d918529a1ea6881482d8ecf3bc9855b534628bc2a7c1: Status 404 returned error can't find the container with id edc4be7eb81dd54b0744d918529a1ea6881482d8ecf3bc9855b534628bc2a7c1 Nov 03 01:29:46.589627 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:46.589598 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-1e406323-6cf7f67fc9-25xxb"] Nov 03 01:29:46.760844 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:29:46.760796 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9844242_b55f_4bc2_9861_50ea42f7bc90.slice/crio-52ee4025d647330a8a8725c427c7c373891fd061ebf13504d266e8a962d0ca56 WatchSource:0}: Error finding container 52ee4025d647330a8a8725c427c7c373891fd061ebf13504d266e8a962d0ca56: Status 404 returned error can't find the container with id 52ee4025d647330a8a8725c427c7c373891fd061ebf13504d266e8a962d0ca56 Nov 03 01:29:46.763085 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:46.763059 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-1e406323-597d58cc5f-qqrhb"] Nov 03 01:29:46.958142 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:46.958115 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-98c7cd16-5799c4dc6c-nz2xm" Nov 03 01:29:46.958335 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:46.958177 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-98c7cd16-d886f8d45-t5dzj" Nov 03 01:29:47.008177 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:47.008075 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-1e406323-597d58cc5f-qqrhb" event={"ID":"e9844242-b55f-4bc2-9861-50ea42f7bc90","Type":"ContainerStarted","Data":"66e9bdb133dd90f8c53eab557b4bfb207f281b57cefe686bdea0c5bde1480d3a"} Nov 03 01:29:47.008177 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:47.008134 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-1e406323-597d58cc5f-qqrhb" event={"ID":"e9844242-b55f-4bc2-9861-50ea42f7bc90","Type":"ContainerStarted","Data":"52ee4025d647330a8a8725c427c7c373891fd061ebf13504d266e8a962d0ca56"} Nov 03 01:29:47.008177 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:47.008165 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-1e406323-597d58cc5f-qqrhb" Nov 03 01:29:47.009867 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:47.009837 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-1e406323-6cf7f67fc9-25xxb" event={"ID":"a932b015-5506-4653-89fb-4a1d9f1c002e","Type":"ContainerStarted","Data":"f4fcc276a00d95dcc88d1243a86cf16678dd754fae681139b1a897de9a49d6cd"} Nov 03 01:29:47.010028 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:47.009893 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-1e406323-6cf7f67fc9-25xxb" event={"ID":"a932b015-5506-4653-89fb-4a1d9f1c002e","Type":"ContainerStarted","Data":"edc4be7eb81dd54b0744d918529a1ea6881482d8ecf3bc9855b534628bc2a7c1"} Nov 03 01:29:47.010028 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:47.009966 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-1e406323-6cf7f67fc9-25xxb" Nov 03 01:29:47.316095 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:47.316015 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-1e406323-597d58cc5f-qqrhb" podStartSLOduration=2.315994126 podStartE2EDuration="2.315994126s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:29:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:29:47.315019172 +0000 UTC m=+669.344501468" watchObservedRunningTime="2026-11-03 01:29:47.315994126 +0000 UTC m=+669.345476425" Nov 03 01:29:47.435149 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:47.435078 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-1e406323-6cf7f67fc9-25xxb" podStartSLOduration=2.435054008 podStartE2EDuration="2.435054008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:29:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:29:47.433807498 +0000 UTC m=+669.463289798" watchObservedRunningTime="2026-11-03 01:29:47.435054008 +0000 UTC m=+669.464536305" Nov 03 01:29:49.076179 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:49.076137 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-21c25193-7c797bdf7-2j4bg"] Nov 03 01:29:49.079751 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:49.079728 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-21c25193-7c797bdf7-2j4bg" Nov 03 01:29:49.135070 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:49.135035 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-21c25193-7c797bdf7-2j4bg"] Nov 03 01:29:49.184665 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:49.184624 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-21c25193-serving-cert\"" Nov 03 01:29:49.219296 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:49.219248 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbkps\" (UniqueName: \"kubernetes.io/projected/6af58ff8-ed74-48e1-935a-50c22f0e1d3f-kube-api-access-zbkps\") pod \"maas-api-e2e-ait-21c25193-7c797bdf7-2j4bg\" (UID: \"6af58ff8-ed74-48e1-935a-50c22f0e1d3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-21c25193-7c797bdf7-2j4bg" Nov 03 01:29:49.219517 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:49.219463 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6af58ff8-ed74-48e1-935a-50c22f0e1d3f-openshift-service-ca\") pod \"maas-api-e2e-ait-21c25193-7c797bdf7-2j4bg\" (UID: \"6af58ff8-ed74-48e1-935a-50c22f0e1d3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-21c25193-7c797bdf7-2j4bg" Nov 03 01:29:49.219597 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:49.219579 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6af58ff8-ed74-48e1-935a-50c22f0e1d3f-metrics-certs\") pod \"maas-api-e2e-ait-21c25193-7c797bdf7-2j4bg\" (UID: \"6af58ff8-ed74-48e1-935a-50c22f0e1d3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-21c25193-7c797bdf7-2j4bg" Nov 03 01:29:49.219637 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:49.219617 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6af58ff8-ed74-48e1-935a-50c22f0e1d3f-maas-api-tls\") pod \"maas-api-e2e-ait-21c25193-7c797bdf7-2j4bg\" (UID: \"6af58ff8-ed74-48e1-935a-50c22f0e1d3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-21c25193-7c797bdf7-2j4bg" Nov 03 01:29:49.318258 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:49.318216 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:49.320401 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:49.320319 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbkps\" (UniqueName: \"kubernetes.io/projected/6af58ff8-ed74-48e1-935a-50c22f0e1d3f-kube-api-access-zbkps\") pod \"maas-api-e2e-ait-21c25193-7c797bdf7-2j4bg\" (UID: \"6af58ff8-ed74-48e1-935a-50c22f0e1d3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-21c25193-7c797bdf7-2j4bg" Nov 03 01:29:49.321323 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:49.320448 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6af58ff8-ed74-48e1-935a-50c22f0e1d3f-openshift-service-ca\") pod \"maas-api-e2e-ait-21c25193-7c797bdf7-2j4bg\" (UID: \"6af58ff8-ed74-48e1-935a-50c22f0e1d3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-21c25193-7c797bdf7-2j4bg" Nov 03 01:29:49.321486 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:49.321404 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6af58ff8-ed74-48e1-935a-50c22f0e1d3f-metrics-certs\") pod \"maas-api-e2e-ait-21c25193-7c797bdf7-2j4bg\" (UID: \"6af58ff8-ed74-48e1-935a-50c22f0e1d3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-21c25193-7c797bdf7-2j4bg" Nov 03 01:29:49.321486 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:49.321440 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6af58ff8-ed74-48e1-935a-50c22f0e1d3f-maas-api-tls\") pod \"maas-api-e2e-ait-21c25193-7c797bdf7-2j4bg\" (UID: \"6af58ff8-ed74-48e1-935a-50c22f0e1d3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-21c25193-7c797bdf7-2j4bg" Nov 03 01:29:49.321671 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:49.321643 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6af58ff8-ed74-48e1-935a-50c22f0e1d3f-openshift-service-ca\") pod \"maas-api-e2e-ait-21c25193-7c797bdf7-2j4bg\" (UID: \"6af58ff8-ed74-48e1-935a-50c22f0e1d3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-21c25193-7c797bdf7-2j4bg" Nov 03 01:29:49.321731 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:29:49.321687 5437 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-21c25193-serving-cert: secret "maas-api-e2e-ait-21c25193-serving-cert" not found Nov 03 01:29:49.321839 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:29:49.321767 5437 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6af58ff8-ed74-48e1-935a-50c22f0e1d3f-maas-api-tls podName:6af58ff8-ed74-48e1-935a-50c22f0e1d3f nodeName:}" failed. No retries permitted until 2026-11-03 01:29:49.821743823 +0000 UTC m=+671.851226118 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6af58ff8-ed74-48e1-935a-50c22f0e1d3f-maas-api-tls") pod "maas-api-e2e-ait-21c25193-7c797bdf7-2j4bg" (UID: "6af58ff8-ed74-48e1-935a-50c22f0e1d3f") : secret "maas-api-e2e-ait-21c25193-serving-cert" not found Nov 03 01:29:49.327186 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:49.327104 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6af58ff8-ed74-48e1-935a-50c22f0e1d3f-metrics-certs\") pod \"maas-api-e2e-ait-21c25193-7c797bdf7-2j4bg\" (UID: \"6af58ff8-ed74-48e1-935a-50c22f0e1d3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-21c25193-7c797bdf7-2j4bg" Nov 03 01:29:49.383233 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:49.383200 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbkps\" (UniqueName: \"kubernetes.io/projected/6af58ff8-ed74-48e1-935a-50c22f0e1d3f-kube-api-access-zbkps\") pod \"maas-api-e2e-ait-21c25193-7c797bdf7-2j4bg\" (UID: \"6af58ff8-ed74-48e1-935a-50c22f0e1d3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-21c25193-7c797bdf7-2j4bg" Nov 03 01:29:49.527197 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:49.527139 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:49.811617 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:49.811575 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:49.825791 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:49.825740 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6af58ff8-ed74-48e1-935a-50c22f0e1d3f-maas-api-tls\") pod \"maas-api-e2e-ait-21c25193-7c797bdf7-2j4bg\" (UID: \"6af58ff8-ed74-48e1-935a-50c22f0e1d3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-21c25193-7c797bdf7-2j4bg" Nov 03 01:29:49.828235 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:49.828198 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6af58ff8-ed74-48e1-935a-50c22f0e1d3f-maas-api-tls\") pod \"maas-api-e2e-ait-21c25193-7c797bdf7-2j4bg\" (UID: \"6af58ff8-ed74-48e1-935a-50c22f0e1d3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-21c25193-7c797bdf7-2j4bg" Nov 03 01:29:49.994051 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:49.994004 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-21c25193-7c797bdf7-2j4bg" Nov 03 01:29:50.029025 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:50.028992 5437 generic.go:356] "Generic (PLEG): container finished" podID="3b980620-e583-42e5-b141-264c2b72bfb2" containerID="bf6c89f4ee5d24971c8e12073f6781f0519e64fc9893b3464598824acda2df83" exitCode=0 Nov 03 01:29:50.029209 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:50.029033 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e3f0a557-gw-openshift-default-7d94d6b5f7-fts5d" event={"ID":"3b980620-e583-42e5-b141-264c2b72bfb2","Type":"ContainerDied","Data":"bf6c89f4ee5d24971c8e12073f6781f0519e64fc9893b3464598824acda2df83"} Nov 03 01:29:50.029597 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:50.029577 5437 scope.go:122] "RemoveContainer" containerID="bf6c89f4ee5d24971c8e12073f6781f0519e64fc9893b3464598824acda2df83" Nov 03 01:29:50.031044 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:50.031006 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 01:29:50.031196 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:50.031087 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 01:29:50.031196 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:50.031122 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 01:29:50.048839 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:29:50.048799 5437 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b980620_e583_42e5_b141_264c2b72bfb2.slice/crio-conmon-bf6c89f4ee5d24971c8e12073f6781f0519e64fc9893b3464598824acda2df83.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b980620_e583_42e5_b141_264c2b72bfb2.slice/crio-bf6c89f4ee5d24971c8e12073f6781f0519e64fc9893b3464598824acda2df83.scope\": RecentStats: unable to find data in memory cache]" Nov 03 01:29:50.377895 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:29:50.377801 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6af58ff8_ed74_48e1_935a_50c22f0e1d3f.slice/crio-57334ed188bde2e4195b73c2e024bcde61d25781caa0ee69beeabe8b32853036 WatchSource:0}: Error finding container 57334ed188bde2e4195b73c2e024bcde61d25781caa0ee69beeabe8b32853036: Status 404 returned error can't find the container with id 57334ed188bde2e4195b73c2e024bcde61d25781caa0ee69beeabe8b32853036 Nov 03 01:29:50.380777 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:50.380727 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-21c25193-7c797bdf7-2j4bg"] Nov 03 01:29:51.049187 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:51.049152 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-21c25193-7c797bdf7-2j4bg" event={"ID":"6af58ff8-ed74-48e1-935a-50c22f0e1d3f","Type":"ContainerStarted","Data":"57334ed188bde2e4195b73c2e024bcde61d25781caa0ee69beeabe8b32853036"} Nov 03 01:29:51.051011 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:51.050986 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e3f0a557-gw-openshift-default-7d94d6b5f7-fts5d" event={"ID":"3b980620-e583-42e5-b141-264c2b72bfb2","Type":"ContainerStarted","Data":"0454264aef2e0df05c3e29e0687cc9086b758405cf68ea077f0b993a65a18e73"} Nov 03 01:29:51.142174 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:51.141724 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-e3f0a557-gw-openshift-default-7d94d6b5f7-fts5d" Nov 03 01:29:51.142832 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:51.142805 5437 patch_prober.go:29] interesting pod/e2e-migrate-e3f0a557-gw-openshift-default-7d94d6b5f7-fts5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Nov 03 01:29:51.142964 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:51.142867 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-e3f0a557-gw-openshift-default-7d94d6b5f7-fts5d" podUID="3b980620-e583-42e5-b141-264c2b72bfb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Nov 03 01:29:51.864931 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:51.864894 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-e3f0a557-gw-openshift-default-7d94d6b5f7-fts5d" Nov 03 01:29:52.058750 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:52.058711 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-21c25193-7c797bdf7-2j4bg" event={"ID":"6af58ff8-ed74-48e1-935a-50c22f0e1d3f","Type":"ContainerStarted","Data":"3e3eb650aa5b4d8df39ece0bc5fc21c04c489ffcae65a99a421ea466d161df63"} Nov 03 01:29:52.058948 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:52.058882 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-21c25193-7c797bdf7-2j4bg" Nov 03 01:29:52.142480 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:52.141864 5437 patch_prober.go:29] interesting pod/e2e-migrate-e3f0a557-gw-openshift-default-7d94d6b5f7-fts5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Nov 03 01:29:52.142480 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:52.141949 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-e3f0a557-gw-openshift-default-7d94d6b5f7-fts5d" podUID="3b980620-e583-42e5-b141-264c2b72bfb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Nov 03 01:29:52.181514 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:52.181458 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-21c25193-7c797bdf7-2j4bg" podStartSLOduration=2.284449403 podStartE2EDuration="3.181441515s" totalImagesPullingTime="896.992112ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:29:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:29:52.180067844 +0000 UTC m=+674.209550138" watchObservedRunningTime="2026-11-03 01:29:52.181441515 +0000 UTC m=+674.210923821" Nov 03 01:29:53.142742 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:53.142697 5437 patch_prober.go:29] interesting pod/e2e-migrate-e3f0a557-gw-openshift-default-7d94d6b5f7-fts5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Nov 03 01:29:53.143156 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:53.142756 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-e3f0a557-gw-openshift-default-7d94d6b5f7-fts5d" podUID="3b980620-e583-42e5-b141-264c2b72bfb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Nov 03 01:29:54.144668 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:54.144122 5437 patch_prober.go:29] interesting pod/e2e-migrate-e3f0a557-gw-openshift-default-7d94d6b5f7-fts5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Nov 03 01:29:54.144668 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:54.144184 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-e3f0a557-gw-openshift-default-7d94d6b5f7-fts5d" podUID="3b980620-e583-42e5-b141-264c2b72bfb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Nov 03 01:29:54.256013 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:54.255973 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:54.493412 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:54.493292 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:55.142816 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:55.142770 5437 patch_prober.go:29] interesting pod/e2e-migrate-e3f0a557-gw-openshift-default-7d94d6b5f7-fts5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Nov 03 01:29:55.143046 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:55.142843 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-e3f0a557-gw-openshift-default-7d94d6b5f7-fts5d" podUID="3b980620-e583-42e5-b141-264c2b72bfb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Nov 03 01:29:56.141718 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:56.141680 5437 patch_prober.go:29] interesting pod/e2e-migrate-e3f0a557-gw-openshift-default-7d94d6b5f7-fts5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Nov 03 01:29:56.142158 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:56.141746 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-e3f0a557-gw-openshift-default-7d94d6b5f7-fts5d" podUID="3b980620-e583-42e5-b141-264c2b72bfb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Nov 03 01:29:57.146204 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:57.146171 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-e3f0a557-gw-openshift-default-7d94d6b5f7-fts5d" Nov 03 01:29:57.147200 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:57.147175 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-e3f0a557-gw-openshift-default-7d94d6b5f7-fts5d" Nov 03 01:29:58.017931 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:58.017896 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-1e406323-6cf7f67fc9-25xxb" Nov 03 01:29:58.018150 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:58.017984 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-1e406323-597d58cc5f-qqrhb" Nov 03 01:29:58.071478 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:58.071447 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-21c25193-7c797bdf7-2j4bg" Nov 03 01:29:59.192298 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:29:59.192252 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:00.177413 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:00.177376 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-21c25193-29894490-7rsm7"] Nov 03 01:30:00.181320 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:00.181296 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-21c25193-29894490-7rsm7" Nov 03 01:30:00.190187 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:00.190147 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-1e406323-29894490-569fs"] Nov 03 01:30:00.193825 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:00.193801 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-1e406323-29894490-569fs" Nov 03 01:30:00.239698 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:00.239663 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fgwsl\"" Nov 03 01:30:00.248215 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:00.248172 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5f8f238-2822-4dc9-a9aa-8b252abc74dc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-21c25193-29894490-7rsm7\" (UID: \"d5f8f238-2822-4dc9-a9aa-8b252abc74dc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-21c25193-29894490-7rsm7" Nov 03 01:30:00.248426 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:00.248298 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztk6v\" (UniqueName: \"kubernetes.io/projected/1a8d248c-1086-47ad-b619-152128cdba1e-kube-api-access-ztk6v\") pod \"maas-api-key-cleanup-e2e-derive-1e406323-29894490-569fs\" (UID: \"1a8d248c-1086-47ad-b619-152128cdba1e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-1e406323-29894490-569fs" Nov 03 01:30:00.248475 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:00.248442 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4sj2\" (UniqueName: \"kubernetes.io/projected/d5f8f238-2822-4dc9-a9aa-8b252abc74dc-kube-api-access-r4sj2\") pod \"maas-api-key-cleanup-e2e-ait-21c25193-29894490-7rsm7\" (UID: \"d5f8f238-2822-4dc9-a9aa-8b252abc74dc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-21c25193-29894490-7rsm7" Nov 03 01:30:00.248510 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:00.248483 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a8d248c-1086-47ad-b619-152128cdba1e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-derive-1e406323-29894490-569fs\" (UID: \"1a8d248c-1086-47ad-b619-152128cdba1e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-1e406323-29894490-569fs" Nov 03 01:30:00.272229 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:00.272188 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-21c25193-29894490-7rsm7"] Nov 03 01:30:00.289037 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:00.288998 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-1e406323-29894490-569fs"] Nov 03 01:30:00.350017 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:00.349972 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4sj2\" (UniqueName: \"kubernetes.io/projected/d5f8f238-2822-4dc9-a9aa-8b252abc74dc-kube-api-access-r4sj2\") pod \"maas-api-key-cleanup-e2e-ait-21c25193-29894490-7rsm7\" (UID: \"d5f8f238-2822-4dc9-a9aa-8b252abc74dc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-21c25193-29894490-7rsm7" Nov 03 01:30:00.350196 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:00.350104 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a8d248c-1086-47ad-b619-152128cdba1e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-derive-1e406323-29894490-569fs\" (UID: \"1a8d248c-1086-47ad-b619-152128cdba1e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-1e406323-29894490-569fs" Nov 03 01:30:00.350196 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:00.350139 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5f8f238-2822-4dc9-a9aa-8b252abc74dc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-21c25193-29894490-7rsm7\" (UID: \"d5f8f238-2822-4dc9-a9aa-8b252abc74dc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-21c25193-29894490-7rsm7" Nov 03 01:30:00.350311 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:00.350235 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztk6v\" (UniqueName: \"kubernetes.io/projected/1a8d248c-1086-47ad-b619-152128cdba1e-kube-api-access-ztk6v\") pod \"maas-api-key-cleanup-e2e-derive-1e406323-29894490-569fs\" (UID: \"1a8d248c-1086-47ad-b619-152128cdba1e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-1e406323-29894490-569fs" Nov 03 01:30:00.351027 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:00.350996 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5f8f238-2822-4dc9-a9aa-8b252abc74dc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-21c25193-29894490-7rsm7\" (UID: \"d5f8f238-2822-4dc9-a9aa-8b252abc74dc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-21c25193-29894490-7rsm7" Nov 03 01:30:00.351169 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:00.351121 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a8d248c-1086-47ad-b619-152128cdba1e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-derive-1e406323-29894490-569fs\" (UID: \"1a8d248c-1086-47ad-b619-152128cdba1e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-1e406323-29894490-569fs" Nov 03 01:30:00.416789 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:00.416747 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztk6v\" (UniqueName: \"kubernetes.io/projected/1a8d248c-1086-47ad-b619-152128cdba1e-kube-api-access-ztk6v\") pod \"maas-api-key-cleanup-e2e-derive-1e406323-29894490-569fs\" (UID: \"1a8d248c-1086-47ad-b619-152128cdba1e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-1e406323-29894490-569fs" Nov 03 01:30:00.416789 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:00.416786 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4sj2\" (UniqueName: \"kubernetes.io/projected/d5f8f238-2822-4dc9-a9aa-8b252abc74dc-kube-api-access-r4sj2\") pod \"maas-api-key-cleanup-e2e-ait-21c25193-29894490-7rsm7\" (UID: \"d5f8f238-2822-4dc9-a9aa-8b252abc74dc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-21c25193-29894490-7rsm7" Nov 03 01:30:00.495071 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:00.494982 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-21c25193-29894490-7rsm7" Nov 03 01:30:00.508774 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:00.508747 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-1e406323-29894490-569fs" Nov 03 01:30:00.874939 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:30:00.874895 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5f8f238_2822_4dc9_a9aa_8b252abc74dc.slice/crio-3c85f238413d46febe3889228ee53abc82b96696c016d745580b0d7ecf53f2f4 WatchSource:0}: Error finding container 3c85f238413d46febe3889228ee53abc82b96696c016d745580b0d7ecf53f2f4: Status 404 returned error can't find the container with id 3c85f238413d46febe3889228ee53abc82b96696c016d745580b0d7ecf53f2f4 Nov 03 01:30:00.877427 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:00.877404 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-21c25193-29894490-7rsm7"] Nov 03 01:30:00.921746 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:30:00.921707 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a8d248c_1086_47ad_b619_152128cdba1e.slice/crio-a53e18be14dfa3b8b8359c10ed90531a4998bfd64cdb3c94eb37e06de9d1f05f WatchSource:0}: Error finding container a53e18be14dfa3b8b8359c10ed90531a4998bfd64cdb3c94eb37e06de9d1f05f: Status 404 returned error can't find the container with id a53e18be14dfa3b8b8359c10ed90531a4998bfd64cdb3c94eb37e06de9d1f05f Nov 03 01:30:00.923870 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:00.923845 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-1e406323-29894490-569fs"] Nov 03 01:30:01.111094 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:01.111052 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-1e406323-29894490-569fs" event={"ID":"1a8d248c-1086-47ad-b619-152128cdba1e","Type":"ContainerStarted","Data":"a53e18be14dfa3b8b8359c10ed90531a4998bfd64cdb3c94eb37e06de9d1f05f"} Nov 03 01:30:01.112518 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:01.112484 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-21c25193-29894490-7rsm7" event={"ID":"d5f8f238-2822-4dc9-a9aa-8b252abc74dc","Type":"ContainerStarted","Data":"3c85f238413d46febe3889228ee53abc82b96696c016d745580b0d7ecf53f2f4"} Nov 03 01:30:03.980543 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:03.980467 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:09.034387 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:09.028419 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:09.170520 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:09.170401 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-1e406323-29894490-569fs" event={"ID":"1a8d248c-1086-47ad-b619-152128cdba1e","Type":"ContainerStarted","Data":"57647e5a9bc4e6e5fc7a557fe3206c271804ea33824b37ff65731ab74732ab00"} Nov 03 01:30:09.172196 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:09.172169 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-21c25193-29894490-7rsm7" event={"ID":"d5f8f238-2822-4dc9-a9aa-8b252abc74dc","Type":"ContainerStarted","Data":"2aed05dd2132b06208aaf2da00c78bbccb1779139bca4a0296d472846a1060d8"} Nov 03 01:30:09.301438 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:09.301380 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-1e406323-29894490-569fs" podStartSLOduration=1.576933293 podStartE2EDuration="9.301351519s" totalImagesPullingTime="7.724418226s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:30:09.300634142 +0000 UTC m=+691.330116430" watchObservedRunningTime="2026-11-03 01:30:09.301351519 +0000 UTC m=+691.330833815" Nov 03 01:30:09.442450 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:09.441720 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-21c25193-29894490-7rsm7" podStartSLOduration=2.204864579 podStartE2EDuration="9.441700967s" totalImagesPullingTime="7.236836388s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:30:09.440714608 +0000 UTC m=+691.470196904" watchObservedRunningTime="2026-11-03 01:30:09.441700967 +0000 UTC m=+691.471183264" Nov 03 01:30:10.179613 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:10.179572 5437 generic.go:356] "Generic (PLEG): container finished" podID="d5f8f238-2822-4dc9-a9aa-8b252abc74dc" containerID="2aed05dd2132b06208aaf2da00c78bbccb1779139bca4a0296d472846a1060d8" exitCode=60 Nov 03 01:30:10.180104 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:10.179660 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-21c25193-29894490-7rsm7" event={"ID":"d5f8f238-2822-4dc9-a9aa-8b252abc74dc","Type":"ContainerDied","Data":"2aed05dd2132b06208aaf2da00c78bbccb1779139bca4a0296d472846a1060d8"} Nov 03 01:30:10.180104 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:10.179885 5437 scope.go:122] "RemoveContainer" containerID="2aed05dd2132b06208aaf2da00c78bbccb1779139bca4a0296d472846a1060d8" Nov 03 01:30:10.181599 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:10.181458 5437 generic.go:356] "Generic (PLEG): container finished" podID="1a8d248c-1086-47ad-b619-152128cdba1e" containerID="57647e5a9bc4e6e5fc7a557fe3206c271804ea33824b37ff65731ab74732ab00" exitCode=60 Nov 03 01:30:10.181599 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:10.181527 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-1e406323-29894490-569fs" event={"ID":"1a8d248c-1086-47ad-b619-152128cdba1e","Type":"ContainerDied","Data":"57647e5a9bc4e6e5fc7a557fe3206c271804ea33824b37ff65731ab74732ab00"} Nov 03 01:30:10.181799 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:10.181784 5437 scope.go:122] "RemoveContainer" containerID="57647e5a9bc4e6e5fc7a557fe3206c271804ea33824b37ff65731ab74732ab00" Nov 03 01:30:11.188454 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:11.188409 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-21c25193-29894490-7rsm7" event={"ID":"d5f8f238-2822-4dc9-a9aa-8b252abc74dc","Type":"ContainerStarted","Data":"cebd02897086c6b186eee49e38060a5714ec5b2bfd9cff56c2ea93b97f961389"} Nov 03 01:30:11.190232 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:11.190202 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-1e406323-29894490-569fs" event={"ID":"1a8d248c-1086-47ad-b619-152128cdba1e","Type":"ContainerStarted","Data":"6029889fc3dddee11bb5c652b60166c261ce8caad47b31ecf5e4d40dedefe4c7"} Nov 03 01:30:12.198845 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:12.198804 5437 generic.go:356] "Generic (PLEG): container finished" podID="1a8d248c-1086-47ad-b619-152128cdba1e" containerID="6029889fc3dddee11bb5c652b60166c261ce8caad47b31ecf5e4d40dedefe4c7" exitCode=60 Nov 03 01:30:12.199311 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:12.198897 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-1e406323-29894490-569fs" event={"ID":"1a8d248c-1086-47ad-b619-152128cdba1e","Type":"ContainerDied","Data":"6029889fc3dddee11bb5c652b60166c261ce8caad47b31ecf5e4d40dedefe4c7"} Nov 03 01:30:12.199311 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:12.198961 5437 scope.go:122] "RemoveContainer" containerID="57647e5a9bc4e6e5fc7a557fe3206c271804ea33824b37ff65731ab74732ab00" Nov 03 01:30:12.199311 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:12.199178 5437 scope.go:122] "RemoveContainer" containerID="6029889fc3dddee11bb5c652b60166c261ce8caad47b31ecf5e4d40dedefe4c7" Nov 03 01:30:12.199510 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:30:12.199469 5437 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-derive-1e406323-29894490-569fs_odh-ai-gateway-infra(1a8d248c-1086-47ad-b619-152128cdba1e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-1e406323-29894490-569fs" podUID="1a8d248c-1086-47ad-b619-152128cdba1e" Nov 03 01:30:12.201670 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:12.201645 5437 generic.go:356] "Generic (PLEG): container finished" podID="d5f8f238-2822-4dc9-a9aa-8b252abc74dc" containerID="cebd02897086c6b186eee49e38060a5714ec5b2bfd9cff56c2ea93b97f961389" exitCode=60 Nov 03 01:30:12.201817 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:12.201732 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-21c25193-29894490-7rsm7" event={"ID":"d5f8f238-2822-4dc9-a9aa-8b252abc74dc","Type":"ContainerDied","Data":"cebd02897086c6b186eee49e38060a5714ec5b2bfd9cff56c2ea93b97f961389"} Nov 03 01:30:12.202036 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:12.202023 5437 scope.go:122] "RemoveContainer" containerID="cebd02897086c6b186eee49e38060a5714ec5b2bfd9cff56c2ea93b97f961389" Nov 03 01:30:12.202285 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:30:12.202262 5437 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-21c25193-29894490-7rsm7_odh-ai-gateway-infra(d5f8f238-2822-4dc9-a9aa-8b252abc74dc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-21c25193-29894490-7rsm7" podUID="d5f8f238-2822-4dc9-a9aa-8b252abc74dc" Nov 03 01:30:12.213688 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:12.213661 5437 scope.go:122] "RemoveContainer" containerID="2aed05dd2132b06208aaf2da00c78bbccb1779139bca4a0296d472846a1060d8" Nov 03 01:30:13.209939 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:13.209904 5437 scope.go:122] "RemoveContainer" containerID="6029889fc3dddee11bb5c652b60166c261ce8caad47b31ecf5e4d40dedefe4c7" Nov 03 01:30:13.210394 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:30:13.210226 5437 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-derive-1e406323-29894490-569fs_odh-ai-gateway-infra(1a8d248c-1086-47ad-b619-152128cdba1e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-1e406323-29894490-569fs" podUID="1a8d248c-1086-47ad-b619-152128cdba1e" Nov 03 01:30:13.211631 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:13.211611 5437 scope.go:122] "RemoveContainer" containerID="cebd02897086c6b186eee49e38060a5714ec5b2bfd9cff56c2ea93b97f961389" Nov 03 01:30:13.211854 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:30:13.211829 5437 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-21c25193-29894490-7rsm7_odh-ai-gateway-infra(d5f8f238-2822-4dc9-a9aa-8b252abc74dc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-21c25193-29894490-7rsm7" podUID="d5f8f238-2822-4dc9-a9aa-8b252abc74dc" Nov 03 01:30:14.035734 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:30:14.035692 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 01:30:14.379305 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:14.379267 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:14.846006 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:14.845963 5437 patch_prober.go:29] interesting pod/e2e-derive-1e406323-openshift-default-5fd6659f69-rs9dq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": read tcp 10.134.0.2:55736->10.134.0.43:15021: read: connection reset by peer" start-of-body= Nov 03 01:30:14.846235 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:14.846057 5437 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-1e406323-openshift-default-5fd6659f69-rs9dq" podUID="576f7646-8212-4ed1-b4f1-2d000cc4072b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": read tcp 10.134.0.2:55736->10.134.0.43:15021: read: connection reset by peer" Nov 03 01:30:15.224620 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:15.224534 5437 generic.go:356] "Generic (PLEG): container finished" podID="576f7646-8212-4ed1-b4f1-2d000cc4072b" containerID="07efcd1440f2d9ab1a3fd5dfa898f5f3fb995fdce8e0091bc90a4dcce598b4e0" exitCode=0 Nov 03 01:30:15.224620 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:15.224604 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-1e406323-openshift-default-5fd6659f69-rs9dq" event={"ID":"576f7646-8212-4ed1-b4f1-2d000cc4072b","Type":"ContainerDied","Data":"07efcd1440f2d9ab1a3fd5dfa898f5f3fb995fdce8e0091bc90a4dcce598b4e0"} Nov 03 01:30:15.224998 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:15.224983 5437 scope.go:122] "RemoveContainer" containerID="07efcd1440f2d9ab1a3fd5dfa898f5f3fb995fdce8e0091bc90a4dcce598b4e0" Nov 03 01:30:15.226325 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:15.226301 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 01:30:15.226409 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:15.226386 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 01:30:15.226456 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:15.226419 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 01:30:16.248789 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:16.248751 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-1e406323-openshift-default-5fd6659f69-rs9dq" event={"ID":"576f7646-8212-4ed1-b4f1-2d000cc4072b","Type":"ContainerStarted","Data":"a3bda85284a34773bbb555652857f2e1ad9c5992c293892ad858589ca28375ba"} Nov 03 01:30:16.640388 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:16.639754 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-1e406323-openshift-default-5fd6659f69-rs9dq" Nov 03 01:30:16.640388 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:16.639888 5437 patch_prober.go:29] interesting pod/e2e-derive-1e406323-openshift-default-5fd6659f69-rs9dq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Nov 03 01:30:16.640388 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:16.639947 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-1e406323-openshift-default-5fd6659f69-rs9dq" podUID="576f7646-8212-4ed1-b4f1-2d000cc4072b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Nov 03 01:30:16.796566 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:16.796527 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc87c747-5xgsf"] Nov 03 01:30:16.803432 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:16.803405 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc87c747-5xgsf" Nov 03 01:30:16.853944 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:16.853903 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc87c747-5xgsf"] Nov 03 01:30:16.858205 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:16.858170 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:30:16.955086 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:16.954982 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8dc484b-86a8-4f73-b762-9d1821a1a335-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-fc87c747-5xgsf\" (UID: \"f8dc484b-86a8-4f73-b762-9d1821a1a335\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc87c747-5xgsf" Nov 03 01:30:16.955086 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:16.955023 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8dc484b-86a8-4f73-b762-9d1821a1a335-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-fc87c747-5xgsf\" (UID: \"f8dc484b-86a8-4f73-b762-9d1821a1a335\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc87c747-5xgsf" Nov 03 01:30:16.955322 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:16.955118 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkhrv\" (UniqueName: \"kubernetes.io/projected/f8dc484b-86a8-4f73-b762-9d1821a1a335-kube-api-access-mkhrv\") pod \"maas-api-revoke-keys-e2e-ait-fc87c747-5xgsf\" (UID: \"f8dc484b-86a8-4f73-b762-9d1821a1a335\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc87c747-5xgsf" Nov 03 01:30:17.056270 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:17.056222 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8dc484b-86a8-4f73-b762-9d1821a1a335-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-fc87c747-5xgsf\" (UID: \"f8dc484b-86a8-4f73-b762-9d1821a1a335\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc87c747-5xgsf" Nov 03 01:30:17.056270 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:17.056269 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8dc484b-86a8-4f73-b762-9d1821a1a335-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-fc87c747-5xgsf\" (UID: \"f8dc484b-86a8-4f73-b762-9d1821a1a335\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc87c747-5xgsf" Nov 03 01:30:17.056636 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:17.056301 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkhrv\" (UniqueName: \"kubernetes.io/projected/f8dc484b-86a8-4f73-b762-9d1821a1a335-kube-api-access-mkhrv\") pod \"maas-api-revoke-keys-e2e-ait-fc87c747-5xgsf\" (UID: \"f8dc484b-86a8-4f73-b762-9d1821a1a335\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc87c747-5xgsf" Nov 03 01:30:17.057055 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:17.057033 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8dc484b-86a8-4f73-b762-9d1821a1a335-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-fc87c747-5xgsf\" (UID: \"f8dc484b-86a8-4f73-b762-9d1821a1a335\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc87c747-5xgsf" Nov 03 01:30:17.058690 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:17.058670 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8dc484b-86a8-4f73-b762-9d1821a1a335-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-fc87c747-5xgsf\" (UID: \"f8dc484b-86a8-4f73-b762-9d1821a1a335\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc87c747-5xgsf" Nov 03 01:30:17.144953 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:17.144922 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkhrv\" (UniqueName: \"kubernetes.io/projected/f8dc484b-86a8-4f73-b762-9d1821a1a335-kube-api-access-mkhrv\") pod \"maas-api-revoke-keys-e2e-ait-fc87c747-5xgsf\" (UID: \"f8dc484b-86a8-4f73-b762-9d1821a1a335\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc87c747-5xgsf" Nov 03 01:30:17.432394 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:17.421574 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc87c747-5xgsf" Nov 03 01:30:17.641391 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:17.640826 5437 patch_prober.go:29] interesting pod/e2e-derive-1e406323-openshift-default-5fd6659f69-rs9dq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Nov 03 01:30:17.641391 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:17.640895 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-1e406323-openshift-default-5fd6659f69-rs9dq" podUID="576f7646-8212-4ed1-b4f1-2d000cc4072b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Nov 03 01:30:17.827612 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:17.825805 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc87c747-5xgsf"] Nov 03 01:30:18.261868 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:18.261834 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc87c747-5xgsf" event={"ID":"f8dc484b-86a8-4f73-b762-9d1821a1a335","Type":"ContainerStarted","Data":"693703fec71b62049ac6a7eda3bddf1624d98b1b543bb7a64f44ecc4edac6670"} Nov 03 01:30:18.262029 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:18.261914 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc87c747-5xgsf" event={"ID":"f8dc484b-86a8-4f73-b762-9d1821a1a335","Type":"ContainerStarted","Data":"c3ce83efdbcd52fb0da85a55579e19fb32e0b707661ae12519f23b2699391b8a"} Nov 03 01:30:18.384457 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:18.384405 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc87c747-5xgsf" podStartSLOduration=2.384392598 podStartE2EDuration="2.384392598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:30:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:30:18.383492757 +0000 UTC m=+700.412975052" watchObservedRunningTime="2026-11-03 01:30:18.384392598 +0000 UTC m=+700.413874893" Nov 03 01:30:18.642455 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:18.636859 5437 patch_prober.go:29] interesting pod/e2e-derive-1e406323-openshift-default-5fd6659f69-rs9dq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Nov 03 01:30:18.642455 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:18.636932 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-1e406323-openshift-default-5fd6659f69-rs9dq" podUID="576f7646-8212-4ed1-b4f1-2d000cc4072b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Nov 03 01:30:19.268701 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:19.268608 5437 generic.go:356] "Generic (PLEG): container finished" podID="f8dc484b-86a8-4f73-b762-9d1821a1a335" containerID="693703fec71b62049ac6a7eda3bddf1624d98b1b543bb7a64f44ecc4edac6670" exitCode=0 Nov 03 01:30:19.268701 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:19.268654 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc87c747-5xgsf" event={"ID":"f8dc484b-86a8-4f73-b762-9d1821a1a335","Type":"ContainerDied","Data":"693703fec71b62049ac6a7eda3bddf1624d98b1b543bb7a64f44ecc4edac6670"} Nov 03 01:30:19.650531 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:19.650496 5437 patch_prober.go:29] interesting pod/e2e-derive-1e406323-openshift-default-5fd6659f69-rs9dq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Nov 03 01:30:19.651052 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:19.650548 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-1e406323-openshift-default-5fd6659f69-rs9dq" podUID="576f7646-8212-4ed1-b4f1-2d000cc4072b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Nov 03 01:30:19.651052 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:19.650980 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:20.459777 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:20.459747 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc87c747-5xgsf" Nov 03 01:30:20.487551 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:20.487501 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8dc484b-86a8-4f73-b762-9d1821a1a335-kube-api-access-mkhrv\" (UniqueName: \"kubernetes.io/projected/f8dc484b-86a8-4f73-b762-9d1821a1a335-kube-api-access-mkhrv\") pod \"f8dc484b-86a8-4f73-b762-9d1821a1a335\" (UID: \"f8dc484b-86a8-4f73-b762-9d1821a1a335\") " Nov 03 01:30:20.487551 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:20.487557 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8dc484b-86a8-4f73-b762-9d1821a1a335-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8dc484b-86a8-4f73-b762-9d1821a1a335-maas-api-openshift-ca\") pod \"f8dc484b-86a8-4f73-b762-9d1821a1a335\" (UID: \"f8dc484b-86a8-4f73-b762-9d1821a1a335\") " Nov 03 01:30:20.487784 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:20.487639 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8dc484b-86a8-4f73-b762-9d1821a1a335-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8dc484b-86a8-4f73-b762-9d1821a1a335-maas-api-xks-ca\") pod \"f8dc484b-86a8-4f73-b762-9d1821a1a335\" (UID: \"f8dc484b-86a8-4f73-b762-9d1821a1a335\") " Nov 03 01:30:20.488044 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:20.488006 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8dc484b-86a8-4f73-b762-9d1821a1a335-maas-api-openshift-ca" pod "f8dc484b-86a8-4f73-b762-9d1821a1a335" (UID: "f8dc484b-86a8-4f73-b762-9d1821a1a335"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:30:20.489945 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:20.489912 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8dc484b-86a8-4f73-b762-9d1821a1a335-kube-api-access-mkhrv" pod "f8dc484b-86a8-4f73-b762-9d1821a1a335" (UID: "f8dc484b-86a8-4f73-b762-9d1821a1a335"). InnerVolumeSpecName "kube-api-access-mkhrv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:30:20.490058 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:20.489947 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8dc484b-86a8-4f73-b762-9d1821a1a335-maas-api-xks-ca" pod "f8dc484b-86a8-4f73-b762-9d1821a1a335" (UID: "f8dc484b-86a8-4f73-b762-9d1821a1a335"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:30:20.589482 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:20.589434 5437 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8dc484b-86a8-4f73-b762-9d1821a1a335-maas-api-xks-ca\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:30:20.589482 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:20.589477 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkhrv\" (UniqueName: \"kubernetes.io/projected/f8dc484b-86a8-4f73-b762-9d1821a1a335-kube-api-access-mkhrv\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:30:20.589482 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:20.589488 5437 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8dc484b-86a8-4f73-b762-9d1821a1a335-maas-api-openshift-ca\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:30:20.642861 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:20.642816 5437 patch_prober.go:29] interesting pod/e2e-derive-1e406323-openshift-default-5fd6659f69-rs9dq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Nov 03 01:30:20.643068 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:20.642892 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-1e406323-openshift-default-5fd6659f69-rs9dq" podUID="576f7646-8212-4ed1-b4f1-2d000cc4072b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Nov 03 01:30:21.280202 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:21.280169 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc87c747-5xgsf" Nov 03 01:30:21.280636 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:21.280205 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc87c747-5xgsf" event={"ID":"f8dc484b-86a8-4f73-b762-9d1821a1a335","Type":"ContainerDied","Data":"c3ce83efdbcd52fb0da85a55579e19fb32e0b707661ae12519f23b2699391b8a"} Nov 03 01:30:21.280636 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:21.280251 5437 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c3ce83efdbcd52fb0da85a55579e19fb32e0b707661ae12519f23b2699391b8a" Nov 03 01:30:21.316846 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:21.316807 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-98c7cd16-d886f8d45-t5dzj"] Nov 03 01:30:21.418104 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:21.418053 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-98c7cd16-5799c4dc6c-nz2xm"] Nov 03 01:30:21.641630 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:21.641600 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-1e406323-openshift-default-5fd6659f69-rs9dq" Nov 03 01:30:21.642004 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:21.641977 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-1e406323-openshift-default-5fd6659f69-rs9dq" Nov 03 01:30:21.642949 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:21.642928 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-1e406323-openshift-default-5fd6659f69-rs9dq" Nov 03 01:30:21.732510 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:21.732471 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:21.934259 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:21.934166 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:23.081739 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:23.081701 5437 scope.go:122] "RemoveContainer" containerID="cebd02897086c6b186eee49e38060a5714ec5b2bfd9cff56c2ea93b97f961389" Nov 03 01:30:24.299909 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:24.299871 5437 generic.go:356] "Generic (PLEG): container finished" podID="d5f8f238-2822-4dc9-a9aa-8b252abc74dc" containerID="48c46142975c310c03880ce5ee5743ecd97a0d0e9da65943b4f5cd6b95e1ba50" exitCode=60 Nov 03 01:30:24.300408 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:24.299914 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-21c25193-29894490-7rsm7" event={"ID":"d5f8f238-2822-4dc9-a9aa-8b252abc74dc","Type":"ContainerDied","Data":"48c46142975c310c03880ce5ee5743ecd97a0d0e9da65943b4f5cd6b95e1ba50"} Nov 03 01:30:24.300408 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:24.299972 5437 scope.go:122] "RemoveContainer" containerID="cebd02897086c6b186eee49e38060a5714ec5b2bfd9cff56c2ea93b97f961389" Nov 03 01:30:24.300408 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:24.300341 5437 scope.go:122] "RemoveContainer" containerID="48c46142975c310c03880ce5ee5743ecd97a0d0e9da65943b4f5cd6b95e1ba50" Nov 03 01:30:24.300696 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:30:24.300675 5437 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-21c25193-29894490-7rsm7_odh-ai-gateway-infra(d5f8f238-2822-4dc9-a9aa-8b252abc74dc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-21c25193-29894490-7rsm7" podUID="d5f8f238-2822-4dc9-a9aa-8b252abc74dc" Nov 03 01:30:24.877042 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:24.877007 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:25.082412 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:25.082348 5437 scope.go:122] "RemoveContainer" containerID="6029889fc3dddee11bb5c652b60166c261ce8caad47b31ecf5e4d40dedefe4c7" Nov 03 01:30:25.428907 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:25.428880 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-21c25193-29894490-7rsm7"] Nov 03 01:30:25.573457 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:25.573430 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-21c25193-29894490-7rsm7" Nov 03 01:30:25.640634 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:25.640534 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5f8f238-2822-4dc9-a9aa-8b252abc74dc-kube-api-access-r4sj2\" (UniqueName: \"kubernetes.io/projected/d5f8f238-2822-4dc9-a9aa-8b252abc74dc-kube-api-access-r4sj2\") pod \"d5f8f238-2822-4dc9-a9aa-8b252abc74dc\" (UID: \"d5f8f238-2822-4dc9-a9aa-8b252abc74dc\") " Nov 03 01:30:25.640817 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:25.640698 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5f8f238-2822-4dc9-a9aa-8b252abc74dc-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5f8f238-2822-4dc9-a9aa-8b252abc74dc-maas-api-service-ca\") pod \"d5f8f238-2822-4dc9-a9aa-8b252abc74dc\" (UID: \"d5f8f238-2822-4dc9-a9aa-8b252abc74dc\") " Nov 03 01:30:25.641045 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:25.641021 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5f8f238-2822-4dc9-a9aa-8b252abc74dc-maas-api-service-ca" pod "d5f8f238-2822-4dc9-a9aa-8b252abc74dc" (UID: "d5f8f238-2822-4dc9-a9aa-8b252abc74dc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:30:25.642668 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:25.642646 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5f8f238-2822-4dc9-a9aa-8b252abc74dc-kube-api-access-r4sj2" pod "d5f8f238-2822-4dc9-a9aa-8b252abc74dc" (UID: "d5f8f238-2822-4dc9-a9aa-8b252abc74dc"). InnerVolumeSpecName "kube-api-access-r4sj2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:30:25.742235 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:25.742193 5437 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5f8f238-2822-4dc9-a9aa-8b252abc74dc-maas-api-service-ca\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:30:25.742235 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:25.742229 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4sj2\" (UniqueName: \"kubernetes.io/projected/d5f8f238-2822-4dc9-a9aa-8b252abc74dc-kube-api-access-r4sj2\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:30:26.317917 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:26.317871 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-98c7cd16-d886f8d45-t5dzj" podUID="2b6e29e8-e823-4a46-82cf-8b3f7720b934" containerName="payload-processing" containerID="cri-o://7c89c98f39aa826e8b69e9cf5e2fe60a9a445e0e13eaee71aca67dcf6bdcb39b" gracePeriod=25 Nov 03 01:30:26.321728 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:26.321697 5437 generic.go:356] "Generic (PLEG): container finished" podID="1a8d248c-1086-47ad-b619-152128cdba1e" containerID="3d3cb91def9d7dc7757627c4775347ba4b7d8ece5f1a6c8b7cc6fd077e9cba55" exitCode=60 Nov 03 01:30:26.321882 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:26.321767 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-1e406323-29894490-569fs" event={"ID":"1a8d248c-1086-47ad-b619-152128cdba1e","Type":"ContainerDied","Data":"3d3cb91def9d7dc7757627c4775347ba4b7d8ece5f1a6c8b7cc6fd077e9cba55"} Nov 03 01:30:26.321882 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:26.321829 5437 scope.go:122] "RemoveContainer" containerID="6029889fc3dddee11bb5c652b60166c261ce8caad47b31ecf5e4d40dedefe4c7" Nov 03 01:30:26.322268 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:26.322239 5437 scope.go:122] "RemoveContainer" containerID="3d3cb91def9d7dc7757627c4775347ba4b7d8ece5f1a6c8b7cc6fd077e9cba55" Nov 03 01:30:26.322641 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:30:26.322583 5437 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-derive-1e406323-29894490-569fs_odh-ai-gateway-infra(1a8d248c-1086-47ad-b619-152128cdba1e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-1e406323-29894490-569fs" podUID="1a8d248c-1086-47ad-b619-152128cdba1e" Nov 03 01:30:26.323518 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:26.323501 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-21c25193-29894490-7rsm7" Nov 03 01:30:26.323606 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:26.323523 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-21c25193-29894490-7rsm7" event={"ID":"d5f8f238-2822-4dc9-a9aa-8b252abc74dc","Type":"ContainerDied","Data":"3c85f238413d46febe3889228ee53abc82b96696c016d745580b0d7ecf53f2f4"} Nov 03 01:30:26.418570 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:26.418523 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-98c7cd16-5799c4dc6c-nz2xm" podUID="9946518f-161b-4ef2-b852-e7be15ed73de" containerName="payload-pre-processing" containerID="cri-o://6ed5b43a6780b2dc6d933ad57420c33772d5941bf45cc5b8fc245d8620e16974" gracePeriod=25 Nov 03 01:30:26.451096 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:26.451061 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-e3f0a557-565f8678bf-lgs4j"] Nov 03 01:30:26.506873 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:26.506849 5437 scope.go:122] "RemoveContainer" containerID="48c46142975c310c03880ce5ee5743ecd97a0d0e9da65943b4f5cd6b95e1ba50" Nov 03 01:30:26.560335 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:26.560299 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-21c25193-29894490-7rsm7"] Nov 03 01:30:26.565838 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:26.565805 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-21c25193-29894490-7rsm7"] Nov 03 01:30:26.587859 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:26.587823 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-98c7cd16-d886f8d45-t5dzj" Nov 03 01:30:26.652112 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:26.652077 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b6e29e8-e823-4a46-82cf-8b3f7720b934-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b6e29e8-e823-4a46-82cf-8b3f7720b934-plugins-config-volume\") pod \"2b6e29e8-e823-4a46-82cf-8b3f7720b934\" (UID: \"2b6e29e8-e823-4a46-82cf-8b3f7720b934\") " Nov 03 01:30:26.652308 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:26.652145 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b6e29e8-e823-4a46-82cf-8b3f7720b934-kube-api-access-d2gmm\" (UniqueName: \"kubernetes.io/projected/2b6e29e8-e823-4a46-82cf-8b3f7720b934-kube-api-access-d2gmm\") pod \"2b6e29e8-e823-4a46-82cf-8b3f7720b934\" (UID: \"2b6e29e8-e823-4a46-82cf-8b3f7720b934\") " Nov 03 01:30:26.652554 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:26.652527 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b6e29e8-e823-4a46-82cf-8b3f7720b934-plugins-config-volume" pod "2b6e29e8-e823-4a46-82cf-8b3f7720b934" (UID: "2b6e29e8-e823-4a46-82cf-8b3f7720b934"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:30:26.654666 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:26.654632 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b6e29e8-e823-4a46-82cf-8b3f7720b934-kube-api-access-d2gmm" pod "2b6e29e8-e823-4a46-82cf-8b3f7720b934" (UID: "2b6e29e8-e823-4a46-82cf-8b3f7720b934"). InnerVolumeSpecName "kube-api-access-d2gmm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:30:26.688578 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:26.688547 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-98c7cd16-5799c4dc6c-nz2xm" Nov 03 01:30:26.753833 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:26.753780 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9946518f-161b-4ef2-b852-e7be15ed73de-kube-api-access-hzthd\" (UniqueName: \"kubernetes.io/projected/9946518f-161b-4ef2-b852-e7be15ed73de-kube-api-access-hzthd\") pod \"9946518f-161b-4ef2-b852-e7be15ed73de\" (UID: \"9946518f-161b-4ef2-b852-e7be15ed73de\") " Nov 03 01:30:26.753833 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:26.753839 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9946518f-161b-4ef2-b852-e7be15ed73de-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9946518f-161b-4ef2-b852-e7be15ed73de-plugins-config-volume\") pod \"9946518f-161b-4ef2-b852-e7be15ed73de\" (UID: \"9946518f-161b-4ef2-b852-e7be15ed73de\") " Nov 03 01:30:26.754231 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:26.754208 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9946518f-161b-4ef2-b852-e7be15ed73de-plugins-config-volume" pod "9946518f-161b-4ef2-b852-e7be15ed73de" (UID: "9946518f-161b-4ef2-b852-e7be15ed73de"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:30:26.754313 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:26.754300 5437 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b6e29e8-e823-4a46-82cf-8b3f7720b934-plugins-config-volume\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:30:26.754348 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:26.754319 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2gmm\" (UniqueName: \"kubernetes.io/projected/2b6e29e8-e823-4a46-82cf-8b3f7720b934-kube-api-access-d2gmm\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:30:26.754348 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:26.754336 5437 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9946518f-161b-4ef2-b852-e7be15ed73de-plugins-config-volume\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:30:26.755955 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:26.755930 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9946518f-161b-4ef2-b852-e7be15ed73de-kube-api-access-hzthd" pod "9946518f-161b-4ef2-b852-e7be15ed73de" (UID: "9946518f-161b-4ef2-b852-e7be15ed73de"). InnerVolumeSpecName "kube-api-access-hzthd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:30:26.859449 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:26.857673 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzthd\" (UniqueName: \"kubernetes.io/projected/9946518f-161b-4ef2-b852-e7be15ed73de-kube-api-access-hzthd\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:30:26.859449 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:26.858268 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:27.112196 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:27.112107 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:27.330493 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:27.330452 5437 generic.go:356] "Generic (PLEG): container finished" podID="2b6e29e8-e823-4a46-82cf-8b3f7720b934" containerID="7c89c98f39aa826e8b69e9cf5e2fe60a9a445e0e13eaee71aca67dcf6bdcb39b" exitCode=0 Nov 03 01:30:27.330681 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:27.330492 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-98c7cd16-d886f8d45-t5dzj" event={"ID":"2b6e29e8-e823-4a46-82cf-8b3f7720b934","Type":"ContainerDied","Data":"7c89c98f39aa826e8b69e9cf5e2fe60a9a445e0e13eaee71aca67dcf6bdcb39b"} Nov 03 01:30:27.330681 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:27.330529 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-98c7cd16-d886f8d45-t5dzj" Nov 03 01:30:27.330681 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:27.330559 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-98c7cd16-d886f8d45-t5dzj" event={"ID":"2b6e29e8-e823-4a46-82cf-8b3f7720b934","Type":"ContainerDied","Data":"98f36588ef1cafb42ffa68569bb991ad7ba9b52c677ba1af2bbaf3bc5d6ea9db"} Nov 03 01:30:27.330681 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:27.330589 5437 scope.go:122] "RemoveContainer" containerID="7c89c98f39aa826e8b69e9cf5e2fe60a9a445e0e13eaee71aca67dcf6bdcb39b" Nov 03 01:30:27.334433 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:27.334405 5437 generic.go:356] "Generic (PLEG): container finished" podID="9946518f-161b-4ef2-b852-e7be15ed73de" containerID="6ed5b43a6780b2dc6d933ad57420c33772d5941bf45cc5b8fc245d8620e16974" exitCode=0 Nov 03 01:30:27.334563 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:27.334491 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-98c7cd16-5799c4dc6c-nz2xm" event={"ID":"9946518f-161b-4ef2-b852-e7be15ed73de","Type":"ContainerDied","Data":"6ed5b43a6780b2dc6d933ad57420c33772d5941bf45cc5b8fc245d8620e16974"} Nov 03 01:30:27.334563 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:27.334542 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-98c7cd16-5799c4dc6c-nz2xm" event={"ID":"9946518f-161b-4ef2-b852-e7be15ed73de","Type":"ContainerDied","Data":"8fd6a70dd0511183153007f5e7c98a373b12bc08ecedb2318c7c351da2a61d6e"} Nov 03 01:30:27.334714 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:27.334624 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-98c7cd16-5799c4dc6c-nz2xm" Nov 03 01:30:27.345854 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:27.345724 5437 scope.go:122] "RemoveContainer" containerID="7c89c98f39aa826e8b69e9cf5e2fe60a9a445e0e13eaee71aca67dcf6bdcb39b" Nov 03 01:30:27.346164 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:30:27.346137 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c89c98f39aa826e8b69e9cf5e2fe60a9a445e0e13eaee71aca67dcf6bdcb39b\": container with ID starting with 7c89c98f39aa826e8b69e9cf5e2fe60a9a445e0e13eaee71aca67dcf6bdcb39b not found: ID does not exist" containerID="7c89c98f39aa826e8b69e9cf5e2fe60a9a445e0e13eaee71aca67dcf6bdcb39b" Nov 03 01:30:27.346256 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:27.346174 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c89c98f39aa826e8b69e9cf5e2fe60a9a445e0e13eaee71aca67dcf6bdcb39b"} err="failed to get container status \"7c89c98f39aa826e8b69e9cf5e2fe60a9a445e0e13eaee71aca67dcf6bdcb39b\": rpc error: code = NotFound desc = could not find container \"7c89c98f39aa826e8b69e9cf5e2fe60a9a445e0e13eaee71aca67dcf6bdcb39b\": container with ID starting with 7c89c98f39aa826e8b69e9cf5e2fe60a9a445e0e13eaee71aca67dcf6bdcb39b not found: ID does not exist" Nov 03 01:30:27.346256 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:27.346200 5437 scope.go:122] "RemoveContainer" containerID="6ed5b43a6780b2dc6d933ad57420c33772d5941bf45cc5b8fc245d8620e16974" Nov 03 01:30:27.355612 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:27.355590 5437 scope.go:122] "RemoveContainer" containerID="6ed5b43a6780b2dc6d933ad57420c33772d5941bf45cc5b8fc245d8620e16974" Nov 03 01:30:27.355981 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:30:27.355964 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ed5b43a6780b2dc6d933ad57420c33772d5941bf45cc5b8fc245d8620e16974\": container with ID starting with 6ed5b43a6780b2dc6d933ad57420c33772d5941bf45cc5b8fc245d8620e16974 not found: ID does not exist" containerID="6ed5b43a6780b2dc6d933ad57420c33772d5941bf45cc5b8fc245d8620e16974" Nov 03 01:30:27.356041 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:27.355990 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ed5b43a6780b2dc6d933ad57420c33772d5941bf45cc5b8fc245d8620e16974"} err="failed to get container status \"6ed5b43a6780b2dc6d933ad57420c33772d5941bf45cc5b8fc245d8620e16974\": rpc error: code = NotFound desc = could not find container \"6ed5b43a6780b2dc6d933ad57420c33772d5941bf45cc5b8fc245d8620e16974\": container with ID starting with 6ed5b43a6780b2dc6d933ad57420c33772d5941bf45cc5b8fc245d8620e16974 not found: ID does not exist" Nov 03 01:30:27.452277 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:27.452184 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-1e406323-29894490-569fs"] Nov 03 01:30:27.499504 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:27.499433 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-98c7cd16-5799c4dc6c-nz2xm"] Nov 03 01:30:27.503324 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:27.503295 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-98c7cd16-5799c4dc6c-nz2xm"] Nov 03 01:30:27.601015 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:27.600985 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-1e406323-29894490-569fs" Nov 03 01:30:27.617681 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:27.617645 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-98c7cd16-d886f8d45-t5dzj"] Nov 03 01:30:27.625051 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:27.625015 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-98c7cd16-d886f8d45-t5dzj"] Nov 03 01:30:27.666826 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:27.666783 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a8d248c-1086-47ad-b619-152128cdba1e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a8d248c-1086-47ad-b619-152128cdba1e-maas-api-service-ca\") pod \"1a8d248c-1086-47ad-b619-152128cdba1e\" (UID: \"1a8d248c-1086-47ad-b619-152128cdba1e\") " Nov 03 01:30:27.667015 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:27.666889 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a8d248c-1086-47ad-b619-152128cdba1e-kube-api-access-ztk6v\" (UniqueName: \"kubernetes.io/projected/1a8d248c-1086-47ad-b619-152128cdba1e-kube-api-access-ztk6v\") pod \"1a8d248c-1086-47ad-b619-152128cdba1e\" (UID: \"1a8d248c-1086-47ad-b619-152128cdba1e\") " Nov 03 01:30:27.667202 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:27.667178 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a8d248c-1086-47ad-b619-152128cdba1e-maas-api-service-ca" pod "1a8d248c-1086-47ad-b619-152128cdba1e" (UID: "1a8d248c-1086-47ad-b619-152128cdba1e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:30:27.667291 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:27.667278 5437 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a8d248c-1086-47ad-b619-152128cdba1e-maas-api-service-ca\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:30:27.669187 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:27.669154 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a8d248c-1086-47ad-b619-152128cdba1e-kube-api-access-ztk6v" pod "1a8d248c-1086-47ad-b619-152128cdba1e" (UID: "1a8d248c-1086-47ad-b619-152128cdba1e"). InnerVolumeSpecName "kube-api-access-ztk6v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:30:27.767926 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:27.767837 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztk6v\" (UniqueName: \"kubernetes.io/projected/1a8d248c-1086-47ad-b619-152128cdba1e-kube-api-access-ztk6v\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:30:28.088646 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:28.088612 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b6e29e8-e823-4a46-82cf-8b3f7720b934" path="/var/lib/kubelet/pods/2b6e29e8-e823-4a46-82cf-8b3f7720b934/volumes" Nov 03 01:30:28.089004 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:28.088991 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9946518f-161b-4ef2-b852-e7be15ed73de" path="/var/lib/kubelet/pods/9946518f-161b-4ef2-b852-e7be15ed73de/volumes" Nov 03 01:30:28.089318 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:28.089307 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5f8f238-2822-4dc9-a9aa-8b252abc74dc" path="/var/lib/kubelet/pods/d5f8f238-2822-4dc9-a9aa-8b252abc74dc/volumes" Nov 03 01:30:28.346183 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:28.346084 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-1e406323-29894490-569fs" Nov 03 01:30:28.346344 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:28.346082 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-1e406323-29894490-569fs" event={"ID":"1a8d248c-1086-47ad-b619-152128cdba1e","Type":"ContainerDied","Data":"a53e18be14dfa3b8b8359c10ed90531a4998bfd64cdb3c94eb37e06de9d1f05f"} Nov 03 01:30:28.346344 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:28.346259 5437 scope.go:122] "RemoveContainer" containerID="3d3cb91def9d7dc7757627c4775347ba4b7d8ece5f1a6c8b7cc6fd077e9cba55" Nov 03 01:30:28.517178 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:28.517143 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-1e406323-29894490-569fs"] Nov 03 01:30:28.568971 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:28.568932 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-1e406323-29894490-569fs"] Nov 03 01:30:29.193635 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:29.193586 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:29.445030 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:29.444931 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:30.089066 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:30.089028 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a8d248c-1086-47ad-b619-152128cdba1e" path="/var/lib/kubelet/pods/1a8d248c-1086-47ad-b619-152128cdba1e/volumes" Nov 03 01:30:30.519181 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:30.519147 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:30.847837 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:30.847795 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:31.128266 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:31.128176 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:31.452279 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:31.452168 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-e3f0a557-565f8678bf-lgs4j" podUID="fd7a749c-b42c-4deb-904b-0bbe6d34ee70" containerName="payload-processing" containerID="cri-o://32bab3f843e4d6da88b7f7d9be093547b1f22c3092fd1e38823067e7b0f9d775" gracePeriod=25 Nov 03 01:30:31.643336 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:31.643297 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:31.715905 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:31.715827 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-e3f0a557-565f8678bf-lgs4j" Nov 03 01:30:31.814126 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:31.814088 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd7a749c-b42c-4deb-904b-0bbe6d34ee70-kube-api-access-n8pf8\" (UniqueName: \"kubernetes.io/projected/fd7a749c-b42c-4deb-904b-0bbe6d34ee70-kube-api-access-n8pf8\") pod \"fd7a749c-b42c-4deb-904b-0bbe6d34ee70\" (UID: \"fd7a749c-b42c-4deb-904b-0bbe6d34ee70\") " Nov 03 01:30:31.814336 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:31.814180 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd7a749c-b42c-4deb-904b-0bbe6d34ee70-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd7a749c-b42c-4deb-904b-0bbe6d34ee70-plugins-config-volume\") pod \"fd7a749c-b42c-4deb-904b-0bbe6d34ee70\" (UID: \"fd7a749c-b42c-4deb-904b-0bbe6d34ee70\") " Nov 03 01:30:31.814596 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:31.814571 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd7a749c-b42c-4deb-904b-0bbe6d34ee70-plugins-config-volume" pod "fd7a749c-b42c-4deb-904b-0bbe6d34ee70" (UID: "fd7a749c-b42c-4deb-904b-0bbe6d34ee70"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:30:31.816319 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:31.816293 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd7a749c-b42c-4deb-904b-0bbe6d34ee70-kube-api-access-n8pf8" pod "fd7a749c-b42c-4deb-904b-0bbe6d34ee70" (UID: "fd7a749c-b42c-4deb-904b-0bbe6d34ee70"). InnerVolumeSpecName "kube-api-access-n8pf8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:30:31.915301 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:31.915260 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8pf8\" (UniqueName: \"kubernetes.io/projected/fd7a749c-b42c-4deb-904b-0bbe6d34ee70-kube-api-access-n8pf8\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:30:31.915301 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:31.915296 5437 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd7a749c-b42c-4deb-904b-0bbe6d34ee70-plugins-config-volume\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:30:32.305762 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:32.305727 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:32.370703 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:32.370619 5437 generic.go:356] "Generic (PLEG): container finished" podID="fd7a749c-b42c-4deb-904b-0bbe6d34ee70" containerID="32bab3f843e4d6da88b7f7d9be093547b1f22c3092fd1e38823067e7b0f9d775" exitCode=0 Nov 03 01:30:32.370703 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:32.370679 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-e3f0a557-565f8678bf-lgs4j" Nov 03 01:30:32.370909 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:32.370703 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-e3f0a557-565f8678bf-lgs4j" event={"ID":"fd7a749c-b42c-4deb-904b-0bbe6d34ee70","Type":"ContainerDied","Data":"32bab3f843e4d6da88b7f7d9be093547b1f22c3092fd1e38823067e7b0f9d775"} Nov 03 01:30:32.370909 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:32.370775 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-e3f0a557-565f8678bf-lgs4j" event={"ID":"fd7a749c-b42c-4deb-904b-0bbe6d34ee70","Type":"ContainerDied","Data":"90e7aef21efffc9fb9cc1f0915aefa26ad8001adf6daee4190e1c10f7b0de43f"} Nov 03 01:30:32.370909 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:32.370797 5437 scope.go:122] "RemoveContainer" containerID="32bab3f843e4d6da88b7f7d9be093547b1f22c3092fd1e38823067e7b0f9d775" Nov 03 01:30:32.379549 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:32.379527 5437 scope.go:122] "RemoveContainer" containerID="32bab3f843e4d6da88b7f7d9be093547b1f22c3092fd1e38823067e7b0f9d775" Nov 03 01:30:32.379867 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:30:32.379842 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32bab3f843e4d6da88b7f7d9be093547b1f22c3092fd1e38823067e7b0f9d775\": container with ID starting with 32bab3f843e4d6da88b7f7d9be093547b1f22c3092fd1e38823067e7b0f9d775 not found: ID does not exist" containerID="32bab3f843e4d6da88b7f7d9be093547b1f22c3092fd1e38823067e7b0f9d775" Nov 03 01:30:32.379918 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:32.379878 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32bab3f843e4d6da88b7f7d9be093547b1f22c3092fd1e38823067e7b0f9d775"} err="failed to get container status \"32bab3f843e4d6da88b7f7d9be093547b1f22c3092fd1e38823067e7b0f9d775\": rpc error: code = NotFound desc = could not find container \"32bab3f843e4d6da88b7f7d9be093547b1f22c3092fd1e38823067e7b0f9d775\": container with ID starting with 32bab3f843e4d6da88b7f7d9be093547b1f22c3092fd1e38823067e7b0f9d775 not found: ID does not exist" Nov 03 01:30:32.540040 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:32.540004 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-e3f0a557-565f8678bf-lgs4j"] Nov 03 01:30:32.585750 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:32.585710 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:32.592200 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:32.592161 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-e3f0a557-565f8678bf-lgs4j"] Nov 03 01:30:32.816013 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:32.815979 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-e3f0a557-gw-openshift-default-7d94d6b5f7-fts5d"] Nov 03 01:30:32.816265 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:32.816239 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-e3f0a557-gw-openshift-default-7d94d6b5f7-fts5d" podUID="3b980620-e583-42e5-b141-264c2b72bfb2" containerName="istio-proxy" containerID="cri-o://0454264aef2e0df05c3e29e0687cc9086b758405cf68ea077f0b993a65a18e73" gracePeriod=30 Nov 03 01:30:32.849616 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:32.849566 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:33.050837 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:33.050799 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:33.352315 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:33.352275 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:33.557020 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:33.556981 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:33.760271 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:33.760173 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:33.821623 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:33.821560 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc87c747-5xgsf"] Nov 03 01:30:33.873182 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:33.873146 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc87c747-5xgsf"] Nov 03 01:30:34.089342 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:34.089310 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8dc484b-86a8-4f73-b762-9d1821a1a335" path="/var/lib/kubelet/pods/f8dc484b-86a8-4f73-b762-9d1821a1a335/volumes" Nov 03 01:30:34.089755 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:34.089738 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd7a749c-b42c-4deb-904b-0bbe6d34ee70" path="/var/lib/kubelet/pods/fd7a749c-b42c-4deb-904b-0bbe6d34ee70/volumes" Nov 03 01:30:34.357397 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:34.357283 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:34.560592 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:34.560555 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:34.801157 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:34.801106 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:35.008547 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:35.008506 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:35.238090 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:35.238001 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:36.116354 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:36.116317 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:36.324898 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:36.324861 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:38.084480 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:38.084396 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-e3f0a557-gw-openshift-default-7d94d6b5f7-fts5d" Nov 03 01:30:38.198425 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:38.198395 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b980620-e583-42e5-b141-264c2b72bfb2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b980620-e583-42e5-b141-264c2b72bfb2-istiod-ca-cert\") pod \"3b980620-e583-42e5-b141-264c2b72bfb2\" (UID: \"3b980620-e583-42e5-b141-264c2b72bfb2\") " Nov 03 01:30:38.198617 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:38.198447 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b980620-e583-42e5-b141-264c2b72bfb2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b980620-e583-42e5-b141-264c2b72bfb2-workload-certs\") pod \"3b980620-e583-42e5-b141-264c2b72bfb2\" (UID: \"3b980620-e583-42e5-b141-264c2b72bfb2\") " Nov 03 01:30:38.198617 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:38.198509 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b980620-e583-42e5-b141-264c2b72bfb2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b980620-e583-42e5-b141-264c2b72bfb2-workload-socket\") pod \"3b980620-e583-42e5-b141-264c2b72bfb2\" (UID: \"3b980620-e583-42e5-b141-264c2b72bfb2\") " Nov 03 01:30:38.198617 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:38.198552 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b980620-e583-42e5-b141-264c2b72bfb2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b980620-e583-42e5-b141-264c2b72bfb2-istio-envoy\") pod \"3b980620-e583-42e5-b141-264c2b72bfb2\" (UID: \"3b980620-e583-42e5-b141-264c2b72bfb2\") " Nov 03 01:30:38.198617 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:38.198587 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b980620-e583-42e5-b141-264c2b72bfb2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b980620-e583-42e5-b141-264c2b72bfb2-credential-socket\") pod \"3b980620-e583-42e5-b141-264c2b72bfb2\" (UID: \"3b980620-e583-42e5-b141-264c2b72bfb2\") " Nov 03 01:30:38.198833 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:38.198617 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b980620-e583-42e5-b141-264c2b72bfb2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b980620-e583-42e5-b141-264c2b72bfb2-istio-data\") pod \"3b980620-e583-42e5-b141-264c2b72bfb2\" (UID: \"3b980620-e583-42e5-b141-264c2b72bfb2\") " Nov 03 01:30:38.198833 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:38.198666 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b980620-e583-42e5-b141-264c2b72bfb2-kube-api-access-48bmb\" (UniqueName: \"kubernetes.io/projected/3b980620-e583-42e5-b141-264c2b72bfb2-kube-api-access-48bmb\") pod \"3b980620-e583-42e5-b141-264c2b72bfb2\" (UID: \"3b980620-e583-42e5-b141-264c2b72bfb2\") " Nov 03 01:30:38.198833 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:38.198708 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3b980620-e583-42e5-b141-264c2b72bfb2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b980620-e583-42e5-b141-264c2b72bfb2-istio-podinfo\") pod \"3b980620-e583-42e5-b141-264c2b72bfb2\" (UID: \"3b980620-e583-42e5-b141-264c2b72bfb2\") " Nov 03 01:30:38.198833 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:38.198703 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b980620-e583-42e5-b141-264c2b72bfb2-workload-certs" pod "3b980620-e583-42e5-b141-264c2b72bfb2" (UID: "3b980620-e583-42e5-b141-264c2b72bfb2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:30:38.198833 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:38.198751 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b980620-e583-42e5-b141-264c2b72bfb2-workload-socket" pod "3b980620-e583-42e5-b141-264c2b72bfb2" (UID: "3b980620-e583-42e5-b141-264c2b72bfb2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:30:38.198833 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:38.198760 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b980620-e583-42e5-b141-264c2b72bfb2-istio-token\" (UniqueName: \"kubernetes.io/projected/3b980620-e583-42e5-b141-264c2b72bfb2-istio-token\") pod \"3b980620-e583-42e5-b141-264c2b72bfb2\" (UID: \"3b980620-e583-42e5-b141-264c2b72bfb2\") " Nov 03 01:30:38.199250 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:38.198897 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b980620-e583-42e5-b141-264c2b72bfb2-istiod-ca-cert" pod "3b980620-e583-42e5-b141-264c2b72bfb2" (UID: "3b980620-e583-42e5-b141-264c2b72bfb2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:30:38.199250 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:38.198905 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b980620-e583-42e5-b141-264c2b72bfb2-credential-socket" pod "3b980620-e583-42e5-b141-264c2b72bfb2" (UID: "3b980620-e583-42e5-b141-264c2b72bfb2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:30:38.199493 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:38.199472 5437 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b980620-e583-42e5-b141-264c2b72bfb2-credential-socket\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:30:38.199716 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:38.199691 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b980620-e583-42e5-b141-264c2b72bfb2-istio-data" pod "3b980620-e583-42e5-b141-264c2b72bfb2" (UID: "3b980620-e583-42e5-b141-264c2b72bfb2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:30:38.199859 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:38.199804 5437 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b980620-e583-42e5-b141-264c2b72bfb2-istiod-ca-cert\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:30:38.199859 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:38.199835 5437 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b980620-e583-42e5-b141-264c2b72bfb2-workload-certs\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:30:38.200035 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:38.199864 5437 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b980620-e583-42e5-b141-264c2b72bfb2-workload-socket\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:30:38.201046 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:38.201022 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3b980620-e583-42e5-b141-264c2b72bfb2-istio-podinfo" pod "3b980620-e583-42e5-b141-264c2b72bfb2" (UID: "3b980620-e583-42e5-b141-264c2b72bfb2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:30:38.201728 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:38.201700 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b980620-e583-42e5-b141-264c2b72bfb2-istio-token" pod "3b980620-e583-42e5-b141-264c2b72bfb2" (UID: "3b980620-e583-42e5-b141-264c2b72bfb2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:30:38.201840 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:38.201747 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b980620-e583-42e5-b141-264c2b72bfb2-istio-envoy" pod "3b980620-e583-42e5-b141-264c2b72bfb2" (UID: "3b980620-e583-42e5-b141-264c2b72bfb2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:30:38.201840 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:38.201758 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b980620-e583-42e5-b141-264c2b72bfb2-kube-api-access-48bmb" pod "3b980620-e583-42e5-b141-264c2b72bfb2" (UID: "3b980620-e583-42e5-b141-264c2b72bfb2"). InnerVolumeSpecName "kube-api-access-48bmb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:30:38.301132 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:38.301090 5437 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b980620-e583-42e5-b141-264c2b72bfb2-istio-envoy\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:30:38.301132 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:38.301125 5437 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b980620-e583-42e5-b141-264c2b72bfb2-istio-data\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:30:38.301132 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:38.301136 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48bmb\" (UniqueName: \"kubernetes.io/projected/3b980620-e583-42e5-b141-264c2b72bfb2-kube-api-access-48bmb\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:30:38.301398 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:38.301145 5437 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b980620-e583-42e5-b141-264c2b72bfb2-istio-podinfo\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:30:38.301398 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:38.301153 5437 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b980620-e583-42e5-b141-264c2b72bfb2-istio-token\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:30:38.404732 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:38.404634 5437 generic.go:356] "Generic (PLEG): container finished" podID="3b980620-e583-42e5-b141-264c2b72bfb2" containerID="0454264aef2e0df05c3e29e0687cc9086b758405cf68ea077f0b993a65a18e73" exitCode=0 Nov 03 01:30:38.404732 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:38.404692 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-e3f0a557-gw-openshift-default-7d94d6b5f7-fts5d" Nov 03 01:30:38.404954 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:38.404728 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e3f0a557-gw-openshift-default-7d94d6b5f7-fts5d" event={"ID":"3b980620-e583-42e5-b141-264c2b72bfb2","Type":"ContainerDied","Data":"0454264aef2e0df05c3e29e0687cc9086b758405cf68ea077f0b993a65a18e73"} Nov 03 01:30:38.404954 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:38.404787 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e3f0a557-gw-openshift-default-7d94d6b5f7-fts5d" event={"ID":"3b980620-e583-42e5-b141-264c2b72bfb2","Type":"ContainerDied","Data":"6dfb85547d6f1397330ce2aaa95c9db74a445e15650693c6724ba06e129a211c"} Nov 03 01:30:38.404954 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:38.404816 5437 scope.go:122] "RemoveContainer" containerID="0454264aef2e0df05c3e29e0687cc9086b758405cf68ea077f0b993a65a18e73" Nov 03 01:30:38.414645 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:38.414621 5437 scope.go:122] "RemoveContainer" containerID="bf6c89f4ee5d24971c8e12073f6781f0519e64fc9893b3464598824acda2df83" Nov 03 01:30:38.424473 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:38.424443 5437 scope.go:122] "RemoveContainer" containerID="0454264aef2e0df05c3e29e0687cc9086b758405cf68ea077f0b993a65a18e73" Nov 03 01:30:38.424835 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:30:38.424812 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0454264aef2e0df05c3e29e0687cc9086b758405cf68ea077f0b993a65a18e73\": container with ID starting with 0454264aef2e0df05c3e29e0687cc9086b758405cf68ea077f0b993a65a18e73 not found: ID does not exist" containerID="0454264aef2e0df05c3e29e0687cc9086b758405cf68ea077f0b993a65a18e73" Nov 03 01:30:38.424947 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:38.424847 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0454264aef2e0df05c3e29e0687cc9086b758405cf68ea077f0b993a65a18e73"} err="failed to get container status \"0454264aef2e0df05c3e29e0687cc9086b758405cf68ea077f0b993a65a18e73\": rpc error: code = NotFound desc = could not find container \"0454264aef2e0df05c3e29e0687cc9086b758405cf68ea077f0b993a65a18e73\": container with ID starting with 0454264aef2e0df05c3e29e0687cc9086b758405cf68ea077f0b993a65a18e73 not found: ID does not exist" Nov 03 01:30:38.424947 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:38.424866 5437 scope.go:122] "RemoveContainer" containerID="bf6c89f4ee5d24971c8e12073f6781f0519e64fc9893b3464598824acda2df83" Nov 03 01:30:38.425147 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:30:38.425130 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf6c89f4ee5d24971c8e12073f6781f0519e64fc9893b3464598824acda2df83\": container with ID starting with bf6c89f4ee5d24971c8e12073f6781f0519e64fc9893b3464598824acda2df83 not found: ID does not exist" containerID="bf6c89f4ee5d24971c8e12073f6781f0519e64fc9893b3464598824acda2df83" Nov 03 01:30:38.425185 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:38.425152 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf6c89f4ee5d24971c8e12073f6781f0519e64fc9893b3464598824acda2df83"} err="failed to get container status \"bf6c89f4ee5d24971c8e12073f6781f0519e64fc9893b3464598824acda2df83\": rpc error: code = NotFound desc = could not find container \"bf6c89f4ee5d24971c8e12073f6781f0519e64fc9893b3464598824acda2df83\": container with ID starting with bf6c89f4ee5d24971c8e12073f6781f0519e64fc9893b3464598824acda2df83 not found: ID does not exist" Nov 03 01:30:38.501995 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:38.501944 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:38.575328 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:38.575288 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-e3f0a557-gw-openshift-default-7d94d6b5f7-fts5d"] Nov 03 01:30:38.583921 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:38.583886 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-e3f0a557-gw-openshift-default-7d94d6b5f7-fts5d"] Nov 03 01:30:38.715283 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:38.715192 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:38.924387 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:38.924327 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:39.123673 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:39.123636 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:39.333387 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:39.333322 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:39.642215 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:39.642183 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:40.093971 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:40.093929 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b980620-e583-42e5-b141-264c2b72bfb2" path="/var/lib/kubelet/pods/3b980620-e583-42e5-b141-264c2b72bfb2/volumes" Nov 03 01:30:41.622212 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:41.622179 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:41.844934 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:41.844899 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:42.632162 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:42.632124 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-21c25193-64474f487f-mndts"] Nov 03 01:30:43.035797 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:43.035561 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:43.284877 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:43.284842 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:43.491601 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:43.491560 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:43.913652 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:43.913613 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:44.040018 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:30:44.039978 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 01:30:47.633549 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:47.633425 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-21c25193-64474f487f-mndts" podUID="ccfd2cf0-8560-48c6-b075-2d5f65f81f5c" containerName="payload-processing" containerID="cri-o://4b051987972220988c216e7b032072c3fdec2efcb67ef54c09e4ab064df8ff7e" gracePeriod=25 Nov 03 01:30:47.888862 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:47.888785 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-21c25193-64474f487f-mndts" Nov 03 01:30:47.899016 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:47.898971 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccfd2cf0-8560-48c6-b075-2d5f65f81f5c-kube-api-access-tdtwv\" (UniqueName: \"kubernetes.io/projected/ccfd2cf0-8560-48c6-b075-2d5f65f81f5c-kube-api-access-tdtwv\") pod \"ccfd2cf0-8560-48c6-b075-2d5f65f81f5c\" (UID: \"ccfd2cf0-8560-48c6-b075-2d5f65f81f5c\") " Nov 03 01:30:47.899188 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:47.899123 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ccfd2cf0-8560-48c6-b075-2d5f65f81f5c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccfd2cf0-8560-48c6-b075-2d5f65f81f5c-plugins-config-volume\") pod \"ccfd2cf0-8560-48c6-b075-2d5f65f81f5c\" (UID: \"ccfd2cf0-8560-48c6-b075-2d5f65f81f5c\") " Nov 03 01:30:47.899501 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:47.899467 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccfd2cf0-8560-48c6-b075-2d5f65f81f5c-plugins-config-volume" pod "ccfd2cf0-8560-48c6-b075-2d5f65f81f5c" (UID: "ccfd2cf0-8560-48c6-b075-2d5f65f81f5c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:30:47.901134 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:47.901096 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccfd2cf0-8560-48c6-b075-2d5f65f81f5c-kube-api-access-tdtwv" pod "ccfd2cf0-8560-48c6-b075-2d5f65f81f5c" (UID: "ccfd2cf0-8560-48c6-b075-2d5f65f81f5c"). InnerVolumeSpecName "kube-api-access-tdtwv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:30:47.999932 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:47.999888 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdtwv\" (UniqueName: \"kubernetes.io/projected/ccfd2cf0-8560-48c6-b075-2d5f65f81f5c-kube-api-access-tdtwv\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:30:47.999932 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:47.999920 5437 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccfd2cf0-8560-48c6-b075-2d5f65f81f5c-plugins-config-volume\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:30:48.460526 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:48.460487 5437 generic.go:356] "Generic (PLEG): container finished" podID="ccfd2cf0-8560-48c6-b075-2d5f65f81f5c" containerID="4b051987972220988c216e7b032072c3fdec2efcb67ef54c09e4ab064df8ff7e" exitCode=0 Nov 03 01:30:48.460742 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:48.460535 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-21c25193-64474f487f-mndts" Nov 03 01:30:48.460742 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:48.460567 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-21c25193-64474f487f-mndts" event={"ID":"ccfd2cf0-8560-48c6-b075-2d5f65f81f5c","Type":"ContainerDied","Data":"4b051987972220988c216e7b032072c3fdec2efcb67ef54c09e4ab064df8ff7e"} Nov 03 01:30:48.460742 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:48.460625 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-21c25193-64474f487f-mndts" event={"ID":"ccfd2cf0-8560-48c6-b075-2d5f65f81f5c","Type":"ContainerDied","Data":"4732a7fba71a4b98859e83e8cd361d8a1c860d8cabaf8ffc81c37012f9158e27"} Nov 03 01:30:48.460742 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:48.460656 5437 scope.go:122] "RemoveContainer" containerID="4b051987972220988c216e7b032072c3fdec2efcb67ef54c09e4ab064df8ff7e" Nov 03 01:30:48.471135 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:48.471109 5437 scope.go:122] "RemoveContainer" containerID="4b051987972220988c216e7b032072c3fdec2efcb67ef54c09e4ab064df8ff7e" Nov 03 01:30:48.471536 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:30:48.471515 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b051987972220988c216e7b032072c3fdec2efcb67ef54c09e4ab064df8ff7e\": container with ID starting with 4b051987972220988c216e7b032072c3fdec2efcb67ef54c09e4ab064df8ff7e not found: ID does not exist" containerID="4b051987972220988c216e7b032072c3fdec2efcb67ef54c09e4ab064df8ff7e" Nov 03 01:30:48.471613 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:48.471548 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b051987972220988c216e7b032072c3fdec2efcb67ef54c09e4ab064df8ff7e"} err="failed to get container status \"4b051987972220988c216e7b032072c3fdec2efcb67ef54c09e4ab064df8ff7e\": rpc error: code = NotFound desc = could not find container \"4b051987972220988c216e7b032072c3fdec2efcb67ef54c09e4ab064df8ff7e\": container with ID starting with 4b051987972220988c216e7b032072c3fdec2efcb67ef54c09e4ab064df8ff7e not found: ID does not exist" Nov 03 01:30:48.577860 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:48.577818 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:48.632626 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:48.632593 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-21c25193-64474f487f-mndts"] Nov 03 01:30:48.634333 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:48.634305 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-21c25193-64474f487f-mndts"] Nov 03 01:30:48.848036 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:48.847994 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:48.917489 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:48.917446 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-0bf34a-66c455cb47-xbxp2"] Nov 03 01:30:48.917926 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:48.917910 5437 cpu_manager.go:414] "containerMap: removing container" podUID="2b6e29e8-e823-4a46-82cf-8b3f7720b934" containerName="payload-processing" Nov 03 01:30:48.917975 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:48.917930 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b6e29e8-e823-4a46-82cf-8b3f7720b934" containerName="payload-processing" Nov 03 01:30:48.917975 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:48.917944 5437 cpu_manager.go:414] "containerMap: removing container" podUID="fd7a749c-b42c-4deb-904b-0bbe6d34ee70" containerName="payload-processing" Nov 03 01:30:48.917975 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:48.917949 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd7a749c-b42c-4deb-904b-0bbe6d34ee70" containerName="payload-processing" Nov 03 01:30:48.917975 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:48.917961 5437 cpu_manager.go:414] "containerMap: removing container" podUID="d5f8f238-2822-4dc9-a9aa-8b252abc74dc" containerName="cleanup" Nov 03 01:30:48.917975 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:48.917966 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5f8f238-2822-4dc9-a9aa-8b252abc74dc" containerName="cleanup" Nov 03 01:30:48.918119 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:48.917977 5437 cpu_manager.go:414] "containerMap: removing container" podUID="3b980620-e583-42e5-b141-264c2b72bfb2" containerName="istio-proxy" Nov 03 01:30:48.918119 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:48.917982 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b980620-e583-42e5-b141-264c2b72bfb2" containerName="istio-proxy" Nov 03 01:30:48.918119 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:48.917991 5437 cpu_manager.go:414] "containerMap: removing container" podUID="3b980620-e583-42e5-b141-264c2b72bfb2" containerName="istio-proxy" Nov 03 01:30:48.918119 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:48.917998 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b980620-e583-42e5-b141-264c2b72bfb2" containerName="istio-proxy" Nov 03 01:30:48.918119 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:48.918008 5437 cpu_manager.go:414] "containerMap: removing container" podUID="d5f8f238-2822-4dc9-a9aa-8b252abc74dc" containerName="cleanup" Nov 03 01:30:48.918119 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:48.918013 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5f8f238-2822-4dc9-a9aa-8b252abc74dc" containerName="cleanup" Nov 03 01:30:48.918119 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:48.918019 5437 cpu_manager.go:414] "containerMap: removing container" podUID="d5f8f238-2822-4dc9-a9aa-8b252abc74dc" containerName="cleanup" Nov 03 01:30:48.918119 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:48.918023 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5f8f238-2822-4dc9-a9aa-8b252abc74dc" containerName="cleanup" Nov 03 01:30:48.918119 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:48.918035 5437 cpu_manager.go:414] "containerMap: removing container" podUID="f8dc484b-86a8-4f73-b762-9d1821a1a335" containerName="revoke-keys" Nov 03 01:30:48.918119 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:48.918039 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8dc484b-86a8-4f73-b762-9d1821a1a335" containerName="revoke-keys" Nov 03 01:30:48.918119 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:48.918045 5437 cpu_manager.go:414] "containerMap: removing container" podUID="1a8d248c-1086-47ad-b619-152128cdba1e" containerName="cleanup" Nov 03 01:30:48.918119 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:48.918049 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a8d248c-1086-47ad-b619-152128cdba1e" containerName="cleanup" Nov 03 01:30:48.918119 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:48.918059 5437 cpu_manager.go:414] "containerMap: removing container" podUID="1a8d248c-1086-47ad-b619-152128cdba1e" containerName="cleanup" Nov 03 01:30:48.918119 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:48.918063 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a8d248c-1086-47ad-b619-152128cdba1e" containerName="cleanup" Nov 03 01:30:48.918119 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:48.918070 5437 cpu_manager.go:414] "containerMap: removing container" podUID="9946518f-161b-4ef2-b852-e7be15ed73de" containerName="payload-pre-processing" Nov 03 01:30:48.918119 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:48.918078 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9946518f-161b-4ef2-b852-e7be15ed73de" containerName="payload-pre-processing" Nov 03 01:30:48.918119 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:48.918090 5437 cpu_manager.go:414] "containerMap: removing container" podUID="1a8d248c-1086-47ad-b619-152128cdba1e" containerName="cleanup" Nov 03 01:30:48.918119 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:48.918094 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a8d248c-1086-47ad-b619-152128cdba1e" containerName="cleanup" Nov 03 01:30:48.918119 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:48.918108 5437 cpu_manager.go:414] "containerMap: removing container" podUID="ccfd2cf0-8560-48c6-b075-2d5f65f81f5c" containerName="payload-processing" Nov 03 01:30:48.918119 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:48.918112 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccfd2cf0-8560-48c6-b075-2d5f65f81f5c" containerName="payload-processing" Nov 03 01:30:48.918704 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:48.918178 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b980620-e583-42e5-b141-264c2b72bfb2" containerName="istio-proxy" Nov 03 01:30:48.918704 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:48.918186 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b980620-e583-42e5-b141-264c2b72bfb2" containerName="istio-proxy" Nov 03 01:30:48.918704 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:48.918191 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5f8f238-2822-4dc9-a9aa-8b252abc74dc" containerName="cleanup" Nov 03 01:30:48.918704 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:48.918196 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5f8f238-2822-4dc9-a9aa-8b252abc74dc" containerName="cleanup" Nov 03 01:30:48.918704 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:48.918203 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8dc484b-86a8-4f73-b762-9d1821a1a335" containerName="revoke-keys" Nov 03 01:30:48.918704 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:48.918209 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a8d248c-1086-47ad-b619-152128cdba1e" containerName="cleanup" Nov 03 01:30:48.918704 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:48.918214 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="9946518f-161b-4ef2-b852-e7be15ed73de" containerName="payload-pre-processing" Nov 03 01:30:48.918704 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:48.918219 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a8d248c-1086-47ad-b619-152128cdba1e" containerName="cleanup" Nov 03 01:30:48.918704 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:48.918225 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccfd2cf0-8560-48c6-b075-2d5f65f81f5c" containerName="payload-processing" Nov 03 01:30:48.918704 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:48.918232 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b6e29e8-e823-4a46-82cf-8b3f7720b934" containerName="payload-processing" Nov 03 01:30:48.918704 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:48.918242 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd7a749c-b42c-4deb-904b-0bbe6d34ee70" containerName="payload-processing" Nov 03 01:30:48.921425 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:48.921401 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0bf34a-66c455cb47-xbxp2" Nov 03 01:30:48.975333 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:48.975292 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-0bf34a-66c455cb47-xbxp2"] Nov 03 01:30:48.976686 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:48.976656 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-0bf34a\"" Nov 03 01:30:48.976869 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:48.976851 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-0bf34a-dockercfg-gfn46\"" Nov 03 01:30:49.008763 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:49.008720 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28d9cb53-e595-4660-9656-0ef00e30d0d4-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-0bf34a-66c455cb47-xbxp2\" (UID: \"28d9cb53-e595-4660-9656-0ef00e30d0d4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0bf34a-66c455cb47-xbxp2" Nov 03 01:30:49.008977 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:49.008788 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfcm8\" (UniqueName: \"kubernetes.io/projected/28d9cb53-e595-4660-9656-0ef00e30d0d4-kube-api-access-hfcm8\") pod \"payload-pre-processing-e2e-worker-w4-0bf34a-66c455cb47-xbxp2\" (UID: \"28d9cb53-e595-4660-9656-0ef00e30d0d4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0bf34a-66c455cb47-xbxp2" Nov 03 01:30:49.109580 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:49.109477 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28d9cb53-e595-4660-9656-0ef00e30d0d4-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-0bf34a-66c455cb47-xbxp2\" (UID: \"28d9cb53-e595-4660-9656-0ef00e30d0d4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0bf34a-66c455cb47-xbxp2" Nov 03 01:30:49.109580 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:49.109566 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfcm8\" (UniqueName: \"kubernetes.io/projected/28d9cb53-e595-4660-9656-0ef00e30d0d4-kube-api-access-hfcm8\") pod \"payload-pre-processing-e2e-worker-w4-0bf34a-66c455cb47-xbxp2\" (UID: \"28d9cb53-e595-4660-9656-0ef00e30d0d4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0bf34a-66c455cb47-xbxp2" Nov 03 01:30:49.110342 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:49.110317 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28d9cb53-e595-4660-9656-0ef00e30d0d4-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-0bf34a-66c455cb47-xbxp2\" (UID: \"28d9cb53-e595-4660-9656-0ef00e30d0d4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0bf34a-66c455cb47-xbxp2" Nov 03 01:30:49.173299 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:49.173264 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfcm8\" (UniqueName: \"kubernetes.io/projected/28d9cb53-e595-4660-9656-0ef00e30d0d4-kube-api-access-hfcm8\") pod \"payload-pre-processing-e2e-worker-w4-0bf34a-66c455cb47-xbxp2\" (UID: \"28d9cb53-e595-4660-9656-0ef00e30d0d4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0bf34a-66c455cb47-xbxp2" Nov 03 01:30:49.183451 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:49.183407 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:49.234444 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:49.234403 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0bf34a-66c455cb47-xbxp2" Nov 03 01:30:49.602921 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:30:49.602849 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28d9cb53_e595_4660_9656_0ef00e30d0d4.slice/crio-30173284b14ae55745dad218452261bc33912a1868e42d98c3729f1881d3e6c7 WatchSource:0}: Error finding container 30173284b14ae55745dad218452261bc33912a1868e42d98c3729f1881d3e6c7: Status 404 returned error can't find the container with id 30173284b14ae55745dad218452261bc33912a1868e42d98c3729f1881d3e6c7 Nov 03 01:30:49.605156 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:49.605121 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-0bf34a-66c455cb47-xbxp2"] Nov 03 01:30:50.093038 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:50.093000 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ccfd2cf0-8560-48c6-b075-2d5f65f81f5c" path="/var/lib/kubelet/pods/ccfd2cf0-8560-48c6-b075-2d5f65f81f5c/volumes" Nov 03 01:30:50.473773 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:50.473669 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0bf34a-66c455cb47-xbxp2" event={"ID":"28d9cb53-e595-4660-9656-0ef00e30d0d4","Type":"ContainerStarted","Data":"1131d4eaeeb8df52191b82138c24643d48e13b6b3dab9cc1546a4675639868ba"} Nov 03 01:30:50.473773 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:50.473738 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0bf34a-66c455cb47-xbxp2" event={"ID":"28d9cb53-e595-4660-9656-0ef00e30d0d4","Type":"ContainerStarted","Data":"30173284b14ae55745dad218452261bc33912a1868e42d98c3729f1881d3e6c7"} Nov 03 01:30:50.473773 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:50.473770 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0bf34a-66c455cb47-xbxp2" Nov 03 01:30:50.592679 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:50.592622 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0bf34a-66c455cb47-xbxp2" podStartSLOduration=2.592608455 podStartE2EDuration="2.592608455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:30:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:30:50.592270262 +0000 UTC m=+732.621752558" watchObservedRunningTime="2026-11-03 01:30:50.592608455 +0000 UTC m=+732.622090750" Nov 03 01:30:50.980433 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:50.980389 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:51.193874 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:51.193833 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:51.312669 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:51.312636 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-b6fb9b-666f59b59f-7d5tv"] Nov 03 01:30:51.313160 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:51.313146 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5f8f238-2822-4dc9-a9aa-8b252abc74dc" containerName="cleanup" Nov 03 01:30:51.313216 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:51.313164 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a8d248c-1086-47ad-b619-152128cdba1e" containerName="cleanup" Nov 03 01:30:51.315113 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:51.315096 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b6fb9b-666f59b59f-7d5tv" Nov 03 01:30:51.327434 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:51.327394 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxkwb\" (UniqueName: \"kubernetes.io/projected/00f0ee4a-6616-454f-8de4-b39189d45442-kube-api-access-xxkwb\") pod \"payload-pre-processing-e2e-worker-w0-b6fb9b-666f59b59f-7d5tv\" (UID: \"00f0ee4a-6616-454f-8de4-b39189d45442\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b6fb9b-666f59b59f-7d5tv" Nov 03 01:30:51.327615 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:51.327496 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00f0ee4a-6616-454f-8de4-b39189d45442-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-b6fb9b-666f59b59f-7d5tv\" (UID: \"00f0ee4a-6616-454f-8de4-b39189d45442\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b6fb9b-666f59b59f-7d5tv" Nov 03 01:30:51.370840 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:51.370799 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-b6fb9b-666f59b59f-7d5tv"] Nov 03 01:30:51.420064 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:51.420036 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-b6fb9b\"" Nov 03 01:30:51.420277 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:51.420257 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-b6fb9b-dockercfg-tnxb7\"" Nov 03 01:30:51.428776 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:51.428737 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00f0ee4a-6616-454f-8de4-b39189d45442-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-b6fb9b-666f59b59f-7d5tv\" (UID: \"00f0ee4a-6616-454f-8de4-b39189d45442\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b6fb9b-666f59b59f-7d5tv" Nov 03 01:30:51.428920 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:51.428905 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxkwb\" (UniqueName: \"kubernetes.io/projected/00f0ee4a-6616-454f-8de4-b39189d45442-kube-api-access-xxkwb\") pod \"payload-pre-processing-e2e-worker-w0-b6fb9b-666f59b59f-7d5tv\" (UID: \"00f0ee4a-6616-454f-8de4-b39189d45442\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b6fb9b-666f59b59f-7d5tv" Nov 03 01:30:51.429531 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:51.429512 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00f0ee4a-6616-454f-8de4-b39189d45442-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-b6fb9b-666f59b59f-7d5tv\" (UID: \"00f0ee4a-6616-454f-8de4-b39189d45442\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b6fb9b-666f59b59f-7d5tv" Nov 03 01:30:51.475275 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:51.475236 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-b6fb9b-8b66f7f77-5xglx"] Nov 03 01:30:51.478579 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:51.478556 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-b6fb9b-8b66f7f77-5xglx" Nov 03 01:30:51.495406 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:51.495351 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxkwb\" (UniqueName: \"kubernetes.io/projected/00f0ee4a-6616-454f-8de4-b39189d45442-kube-api-access-xxkwb\") pod \"payload-pre-processing-e2e-worker-w0-b6fb9b-666f59b59f-7d5tv\" (UID: \"00f0ee4a-6616-454f-8de4-b39189d45442\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b6fb9b-666f59b59f-7d5tv" Nov 03 01:30:51.529825 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:51.529787 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llld6\" (UniqueName: \"kubernetes.io/projected/0c2adf4b-1195-4937-b7d1-ea164abf40ab-kube-api-access-llld6\") pod \"payload-processing-e2e-worker-w0-b6fb9b-8b66f7f77-5xglx\" (UID: \"0c2adf4b-1195-4937-b7d1-ea164abf40ab\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-b6fb9b-8b66f7f77-5xglx" Nov 03 01:30:51.530136 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:51.529851 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c2adf4b-1195-4937-b7d1-ea164abf40ab-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-b6fb9b-8b66f7f77-5xglx\" (UID: \"0c2adf4b-1195-4937-b7d1-ea164abf40ab\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-b6fb9b-8b66f7f77-5xglx" Nov 03 01:30:51.532586 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:51.532551 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-b6fb9b-8b66f7f77-5xglx"] Nov 03 01:30:51.627349 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:51.627263 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b6fb9b-666f59b59f-7d5tv" Nov 03 01:30:51.630551 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:51.630528 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c2adf4b-1195-4937-b7d1-ea164abf40ab-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-b6fb9b-8b66f7f77-5xglx\" (UID: \"0c2adf4b-1195-4937-b7d1-ea164abf40ab\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-b6fb9b-8b66f7f77-5xglx" Nov 03 01:30:51.630669 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:51.630653 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llld6\" (UniqueName: \"kubernetes.io/projected/0c2adf4b-1195-4937-b7d1-ea164abf40ab-kube-api-access-llld6\") pod \"payload-processing-e2e-worker-w0-b6fb9b-8b66f7f77-5xglx\" (UID: \"0c2adf4b-1195-4937-b7d1-ea164abf40ab\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-b6fb9b-8b66f7f77-5xglx" Nov 03 01:30:51.631517 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:51.631493 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c2adf4b-1195-4937-b7d1-ea164abf40ab-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-b6fb9b-8b66f7f77-5xglx\" (UID: \"0c2adf4b-1195-4937-b7d1-ea164abf40ab\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-b6fb9b-8b66f7f77-5xglx" Nov 03 01:30:51.694777 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:51.694742 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-llld6\" (UniqueName: \"kubernetes.io/projected/0c2adf4b-1195-4937-b7d1-ea164abf40ab-kube-api-access-llld6\") pod \"payload-processing-e2e-worker-w0-b6fb9b-8b66f7f77-5xglx\" (UID: \"0c2adf4b-1195-4937-b7d1-ea164abf40ab\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-b6fb9b-8b66f7f77-5xglx" Nov 03 01:30:51.791342 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:51.791301 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-b6fb9b-8b66f7f77-5xglx" Nov 03 01:30:51.923983 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:51.923881 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-1e406323-597d58cc5f-qqrhb"] Nov 03 01:30:52.027032 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:52.026962 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-1e406323-6cf7f67fc9-25xxb"] Nov 03 01:30:52.028323 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:30:52.028299 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00f0ee4a_6616_454f_8de4_b39189d45442.slice/crio-e08b7bdd9c1ebd370ca3ac50366c667edcb7959fe167fffca78992aa7fdbef1b WatchSource:0}: Error finding container e08b7bdd9c1ebd370ca3ac50366c667edcb7959fe167fffca78992aa7fdbef1b: Status 404 returned error can't find the container with id e08b7bdd9c1ebd370ca3ac50366c667edcb7959fe167fffca78992aa7fdbef1b Nov 03 01:30:52.029301 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:52.029277 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-b6fb9b-666f59b59f-7d5tv"] Nov 03 01:30:52.160650 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:30:52.160587 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c2adf4b_1195_4937_b7d1_ea164abf40ab.slice/crio-14db8b36c8647b14086b7fe28acd4e7aca87ceca9d6f956968d47eabbff0c311 WatchSource:0}: Error finding container 14db8b36c8647b14086b7fe28acd4e7aca87ceca9d6f956968d47eabbff0c311: Status 404 returned error can't find the container with id 14db8b36c8647b14086b7fe28acd4e7aca87ceca9d6f956968d47eabbff0c311 Nov 03 01:30:52.163318 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:52.163287 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-b6fb9b-8b66f7f77-5xglx"] Nov 03 01:30:52.339947 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:52.339907 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:52.493209 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:52.493106 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-b6fb9b-8b66f7f77-5xglx" event={"ID":"0c2adf4b-1195-4937-b7d1-ea164abf40ab","Type":"ContainerStarted","Data":"d6e3091dbb32da201a31c0c68e925ce035764de85860a377f0d18c5056e1d8c5"} Nov 03 01:30:52.493209 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:52.493191 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-b6fb9b-8b66f7f77-5xglx" Nov 03 01:30:52.493475 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:52.493219 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-b6fb9b-8b66f7f77-5xglx" event={"ID":"0c2adf4b-1195-4937-b7d1-ea164abf40ab","Type":"ContainerStarted","Data":"14db8b36c8647b14086b7fe28acd4e7aca87ceca9d6f956968d47eabbff0c311"} Nov 03 01:30:52.494862 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:52.494829 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b6fb9b-666f59b59f-7d5tv" event={"ID":"00f0ee4a-6616-454f-8de4-b39189d45442","Type":"ContainerStarted","Data":"8d8aaa5019ad072a5d9e32b8c7f6dbd0a4d19b4f0b384ce6f58a656bbfa29ff7"} Nov 03 01:30:52.494993 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:52.494887 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b6fb9b-666f59b59f-7d5tv" event={"ID":"00f0ee4a-6616-454f-8de4-b39189d45442","Type":"ContainerStarted","Data":"e08b7bdd9c1ebd370ca3ac50366c667edcb7959fe167fffca78992aa7fdbef1b"} Nov 03 01:30:52.494993 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:52.494941 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b6fb9b-666f59b59f-7d5tv" Nov 03 01:30:52.589470 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:52.589430 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:52.613920 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:52.613849 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-b6fb9b-8b66f7f77-5xglx" podStartSLOduration=1.613834049 podStartE2EDuration="1.613834049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:30:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:30:52.612642933 +0000 UTC m=+734.642125228" watchObservedRunningTime="2026-11-03 01:30:52.613834049 +0000 UTC m=+734.643316336" Nov 03 01:30:52.732820 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:52.732765 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b6fb9b-666f59b59f-7d5tv" podStartSLOduration=1.732750213 podStartE2EDuration="1.732750213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:30:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:30:52.731902525 +0000 UTC m=+734.761384822" watchObservedRunningTime="2026-11-03 01:30:52.732750213 +0000 UTC m=+734.762232554" Nov 03 01:30:54.134651 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:54.134613 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:54.340119 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:54.340083 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:54.543916 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:54.543815 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:56.924747 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:56.924687 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-1e406323-597d58cc5f-qqrhb" podUID="e9844242-b55f-4bc2-9861-50ea42f7bc90" containerName="payload-processing" containerID="cri-o://66e9bdb133dd90f8c53eab557b4bfb207f281b57cefe686bdea0c5bde1480d3a" gracePeriod=25 Nov 03 01:30:57.027985 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:57.027916 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-1e406323-6cf7f67fc9-25xxb" podUID="a932b015-5506-4653-89fb-4a1d9f1c002e" containerName="payload-pre-processing" containerID="cri-o://f4fcc276a00d95dcc88d1243a86cf16678dd754fae681139b1a897de9a49d6cd" gracePeriod=25 Nov 03 01:30:57.189785 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:57.189720 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-1e406323-597d58cc5f-qqrhb" Nov 03 01:30:57.287448 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:57.287418 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9844242-b55f-4bc2-9861-50ea42f7bc90-kube-api-access-f8hvd\" (UniqueName: \"kubernetes.io/projected/e9844242-b55f-4bc2-9861-50ea42f7bc90-kube-api-access-f8hvd\") pod \"e9844242-b55f-4bc2-9861-50ea42f7bc90\" (UID: \"e9844242-b55f-4bc2-9861-50ea42f7bc90\") " Nov 03 01:30:57.287611 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:57.287555 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9844242-b55f-4bc2-9861-50ea42f7bc90-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9844242-b55f-4bc2-9861-50ea42f7bc90-plugins-config-volume\") pod \"e9844242-b55f-4bc2-9861-50ea42f7bc90\" (UID: \"e9844242-b55f-4bc2-9861-50ea42f7bc90\") " Nov 03 01:30:57.288127 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:57.288092 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9844242-b55f-4bc2-9861-50ea42f7bc90-plugins-config-volume" pod "e9844242-b55f-4bc2-9861-50ea42f7bc90" (UID: "e9844242-b55f-4bc2-9861-50ea42f7bc90"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:30:57.290213 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:57.290119 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9844242-b55f-4bc2-9861-50ea42f7bc90-kube-api-access-f8hvd" pod "e9844242-b55f-4bc2-9861-50ea42f7bc90" (UID: "e9844242-b55f-4bc2-9861-50ea42f7bc90"). InnerVolumeSpecName "kube-api-access-f8hvd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:30:57.300805 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:57.300770 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-1e406323-6cf7f67fc9-25xxb" Nov 03 01:30:57.388197 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:57.388164 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a932b015-5506-4653-89fb-4a1d9f1c002e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a932b015-5506-4653-89fb-4a1d9f1c002e-plugins-config-volume\") pod \"a932b015-5506-4653-89fb-4a1d9f1c002e\" (UID: \"a932b015-5506-4653-89fb-4a1d9f1c002e\") " Nov 03 01:30:57.388428 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:57.388240 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a932b015-5506-4653-89fb-4a1d9f1c002e-kube-api-access-d99vp\" (UniqueName: \"kubernetes.io/projected/a932b015-5506-4653-89fb-4a1d9f1c002e-kube-api-access-d99vp\") pod \"a932b015-5506-4653-89fb-4a1d9f1c002e\" (UID: \"a932b015-5506-4653-89fb-4a1d9f1c002e\") " Nov 03 01:30:57.388584 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:57.388556 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a932b015-5506-4653-89fb-4a1d9f1c002e-plugins-config-volume" pod "a932b015-5506-4653-89fb-4a1d9f1c002e" (UID: "a932b015-5506-4653-89fb-4a1d9f1c002e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:30:57.388682 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:57.388626 5437 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9844242-b55f-4bc2-9861-50ea42f7bc90-plugins-config-volume\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:30:57.388682 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:57.388643 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8hvd\" (UniqueName: \"kubernetes.io/projected/e9844242-b55f-4bc2-9861-50ea42f7bc90-kube-api-access-f8hvd\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:30:57.390519 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:57.390494 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a932b015-5506-4653-89fb-4a1d9f1c002e-kube-api-access-d99vp" pod "a932b015-5506-4653-89fb-4a1d9f1c002e" (UID: "a932b015-5506-4653-89fb-4a1d9f1c002e"). InnerVolumeSpecName "kube-api-access-d99vp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:30:57.490207 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:57.490076 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d99vp\" (UniqueName: \"kubernetes.io/projected/a932b015-5506-4653-89fb-4a1d9f1c002e-kube-api-access-d99vp\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:30:57.490207 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:57.490111 5437 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a932b015-5506-4653-89fb-4a1d9f1c002e-plugins-config-volume\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:30:57.524472 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:57.524435 5437 generic.go:356] "Generic (PLEG): container finished" podID="a932b015-5506-4653-89fb-4a1d9f1c002e" containerID="f4fcc276a00d95dcc88d1243a86cf16678dd754fae681139b1a897de9a49d6cd" exitCode=0 Nov 03 01:30:57.524688 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:57.524472 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-1e406323-6cf7f67fc9-25xxb" event={"ID":"a932b015-5506-4653-89fb-4a1d9f1c002e","Type":"ContainerDied","Data":"f4fcc276a00d95dcc88d1243a86cf16678dd754fae681139b1a897de9a49d6cd"} Nov 03 01:30:57.524688 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:57.524509 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-1e406323-6cf7f67fc9-25xxb" Nov 03 01:30:57.524688 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:57.524613 5437 scope.go:122] "RemoveContainer" containerID="f4fcc276a00d95dcc88d1243a86cf16678dd754fae681139b1a897de9a49d6cd" Nov 03 01:30:57.524879 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:57.524854 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-1e406323-6cf7f67fc9-25xxb" event={"ID":"a932b015-5506-4653-89fb-4a1d9f1c002e","Type":"ContainerDied","Data":"edc4be7eb81dd54b0744d918529a1ea6881482d8ecf3bc9855b534628bc2a7c1"} Nov 03 01:30:57.526147 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:57.526123 5437 generic.go:356] "Generic (PLEG): container finished" podID="e9844242-b55f-4bc2-9861-50ea42f7bc90" containerID="66e9bdb133dd90f8c53eab557b4bfb207f281b57cefe686bdea0c5bde1480d3a" exitCode=0 Nov 03 01:30:57.526250 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:57.526180 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-1e406323-597d58cc5f-qqrhb" Nov 03 01:30:57.526250 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:57.526205 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-1e406323-597d58cc5f-qqrhb" event={"ID":"e9844242-b55f-4bc2-9861-50ea42f7bc90","Type":"ContainerDied","Data":"66e9bdb133dd90f8c53eab557b4bfb207f281b57cefe686bdea0c5bde1480d3a"} Nov 03 01:30:57.526344 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:57.526255 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-1e406323-597d58cc5f-qqrhb" event={"ID":"e9844242-b55f-4bc2-9861-50ea42f7bc90","Type":"ContainerDied","Data":"52ee4025d647330a8a8725c427c7c373891fd061ebf13504d266e8a962d0ca56"} Nov 03 01:30:57.545554 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:57.545528 5437 scope.go:122] "RemoveContainer" containerID="f4fcc276a00d95dcc88d1243a86cf16678dd754fae681139b1a897de9a49d6cd" Nov 03 01:30:57.545912 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:30:57.545887 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4fcc276a00d95dcc88d1243a86cf16678dd754fae681139b1a897de9a49d6cd\": container with ID starting with f4fcc276a00d95dcc88d1243a86cf16678dd754fae681139b1a897de9a49d6cd not found: ID does not exist" containerID="f4fcc276a00d95dcc88d1243a86cf16678dd754fae681139b1a897de9a49d6cd" Nov 03 01:30:57.545996 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:57.545920 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4fcc276a00d95dcc88d1243a86cf16678dd754fae681139b1a897de9a49d6cd"} err="failed to get container status \"f4fcc276a00d95dcc88d1243a86cf16678dd754fae681139b1a897de9a49d6cd\": rpc error: code = NotFound desc = could not find container \"f4fcc276a00d95dcc88d1243a86cf16678dd754fae681139b1a897de9a49d6cd\": container with ID starting with f4fcc276a00d95dcc88d1243a86cf16678dd754fae681139b1a897de9a49d6cd not found: ID does not exist" Nov 03 01:30:57.545996 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:57.545940 5437 scope.go:122] "RemoveContainer" containerID="66e9bdb133dd90f8c53eab557b4bfb207f281b57cefe686bdea0c5bde1480d3a" Nov 03 01:30:57.554788 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:57.554761 5437 scope.go:122] "RemoveContainer" containerID="66e9bdb133dd90f8c53eab557b4bfb207f281b57cefe686bdea0c5bde1480d3a" Nov 03 01:30:57.555129 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:30:57.555107 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66e9bdb133dd90f8c53eab557b4bfb207f281b57cefe686bdea0c5bde1480d3a\": container with ID starting with 66e9bdb133dd90f8c53eab557b4bfb207f281b57cefe686bdea0c5bde1480d3a not found: ID does not exist" containerID="66e9bdb133dd90f8c53eab557b4bfb207f281b57cefe686bdea0c5bde1480d3a" Nov 03 01:30:57.555194 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:57.555139 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66e9bdb133dd90f8c53eab557b4bfb207f281b57cefe686bdea0c5bde1480d3a"} err="failed to get container status \"66e9bdb133dd90f8c53eab557b4bfb207f281b57cefe686bdea0c5bde1480d3a\": rpc error: code = NotFound desc = could not find container \"66e9bdb133dd90f8c53eab557b4bfb207f281b57cefe686bdea0c5bde1480d3a\": container with ID starting with 66e9bdb133dd90f8c53eab557b4bfb207f281b57cefe686bdea0c5bde1480d3a not found: ID does not exist" Nov 03 01:30:57.693707 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:57.693667 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-1e406323-6cf7f67fc9-25xxb"] Nov 03 01:30:57.697391 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:57.697348 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-1e406323-6cf7f67fc9-25xxb"] Nov 03 01:30:57.809638 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:57.809599 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-1e406323-597d58cc5f-qqrhb"] Nov 03 01:30:57.812987 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:57.812957 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-1e406323-597d58cc5f-qqrhb"] Nov 03 01:30:58.088675 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:58.088595 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a932b015-5506-4653-89fb-4a1d9f1c002e" path="/var/lib/kubelet/pods/a932b015-5506-4653-89fb-4a1d9f1c002e/volumes" Nov 03 01:30:58.089038 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:58.088976 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9844242-b55f-4bc2-9861-50ea42f7bc90" path="/var/lib/kubelet/pods/e9844242-b55f-4bc2-9861-50ea42f7bc90/volumes" Nov 03 01:30:59.174536 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:59.174495 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-0bf34a-754fc9fc-tx5ff"] Nov 03 01:30:59.174958 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:59.174943 5437 cpu_manager.go:414] "containerMap: removing container" podUID="e9844242-b55f-4bc2-9861-50ea42f7bc90" containerName="payload-processing" Nov 03 01:30:59.174958 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:59.174955 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9844242-b55f-4bc2-9861-50ea42f7bc90" containerName="payload-processing" Nov 03 01:30:59.175061 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:59.174978 5437 cpu_manager.go:414] "containerMap: removing container" podUID="a932b015-5506-4653-89fb-4a1d9f1c002e" containerName="payload-pre-processing" Nov 03 01:30:59.175061 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:59.174985 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a932b015-5506-4653-89fb-4a1d9f1c002e" containerName="payload-pre-processing" Nov 03 01:30:59.175061 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:59.175061 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="a932b015-5506-4653-89fb-4a1d9f1c002e" containerName="payload-pre-processing" Nov 03 01:30:59.175217 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:59.175072 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9844242-b55f-4bc2-9861-50ea42f7bc90" containerName="payload-processing" Nov 03 01:30:59.178118 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:59.178094 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0bf34a-754fc9fc-tx5ff" Nov 03 01:30:59.206963 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:59.206922 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f9f6115-560b-451b-ba0d-e66e67ca55eb-metrics-certs\") pod \"maas-api-e2e-worker-w4-0bf34a-754fc9fc-tx5ff\" (UID: \"1f9f6115-560b-451b-ba0d-e66e67ca55eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0bf34a-754fc9fc-tx5ff" Nov 03 01:30:59.207145 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:59.206977 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f9f6115-560b-451b-ba0d-e66e67ca55eb-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-0bf34a-754fc9fc-tx5ff\" (UID: \"1f9f6115-560b-451b-ba0d-e66e67ca55eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0bf34a-754fc9fc-tx5ff" Nov 03 01:30:59.207145 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:59.207088 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f9f6115-560b-451b-ba0d-e66e67ca55eb-maas-api-tls\") pod \"maas-api-e2e-worker-w4-0bf34a-754fc9fc-tx5ff\" (UID: \"1f9f6115-560b-451b-ba0d-e66e67ca55eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0bf34a-754fc9fc-tx5ff" Nov 03 01:30:59.207221 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:59.207161 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cngdj\" (UniqueName: \"kubernetes.io/projected/1f9f6115-560b-451b-ba0d-e66e67ca55eb-kube-api-access-cngdj\") pod \"maas-api-e2e-worker-w4-0bf34a-754fc9fc-tx5ff\" (UID: \"1f9f6115-560b-451b-ba0d-e66e67ca55eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0bf34a-754fc9fc-tx5ff" Nov 03 01:30:59.233707 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:59.233669 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-0bf34a-754fc9fc-tx5ff"] Nov 03 01:30:59.283312 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:59.283280 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-0bf34a-serving-cert\"" Nov 03 01:30:59.308040 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:59.308000 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f9f6115-560b-451b-ba0d-e66e67ca55eb-metrics-certs\") pod \"maas-api-e2e-worker-w4-0bf34a-754fc9fc-tx5ff\" (UID: \"1f9f6115-560b-451b-ba0d-e66e67ca55eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0bf34a-754fc9fc-tx5ff" Nov 03 01:30:59.308178 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:59.308064 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f9f6115-560b-451b-ba0d-e66e67ca55eb-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-0bf34a-754fc9fc-tx5ff\" (UID: \"1f9f6115-560b-451b-ba0d-e66e67ca55eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0bf34a-754fc9fc-tx5ff" Nov 03 01:30:59.308178 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:59.308094 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f9f6115-560b-451b-ba0d-e66e67ca55eb-maas-api-tls\") pod \"maas-api-e2e-worker-w4-0bf34a-754fc9fc-tx5ff\" (UID: \"1f9f6115-560b-451b-ba0d-e66e67ca55eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0bf34a-754fc9fc-tx5ff" Nov 03 01:30:59.308178 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:59.308146 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cngdj\" (UniqueName: \"kubernetes.io/projected/1f9f6115-560b-451b-ba0d-e66e67ca55eb-kube-api-access-cngdj\") pod \"maas-api-e2e-worker-w4-0bf34a-754fc9fc-tx5ff\" (UID: \"1f9f6115-560b-451b-ba0d-e66e67ca55eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0bf34a-754fc9fc-tx5ff" Nov 03 01:30:59.308310 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:30:59.308278 5437 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w4-0bf34a-serving-cert: secret "maas-api-e2e-worker-w4-0bf34a-serving-cert" not found Nov 03 01:30:59.308394 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:30:59.308359 5437 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f9f6115-560b-451b-ba0d-e66e67ca55eb-maas-api-tls podName:1f9f6115-560b-451b-ba0d-e66e67ca55eb nodeName:}" failed. No retries permitted until 2026-11-03 01:30:59.808335917 +0000 UTC m=+741.837818197 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1f9f6115-560b-451b-ba0d-e66e67ca55eb-maas-api-tls") pod "maas-api-e2e-worker-w4-0bf34a-754fc9fc-tx5ff" (UID: "1f9f6115-560b-451b-ba0d-e66e67ca55eb") : secret "maas-api-e2e-worker-w4-0bf34a-serving-cert" not found Nov 03 01:30:59.308823 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:59.308804 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f9f6115-560b-451b-ba0d-e66e67ca55eb-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-0bf34a-754fc9fc-tx5ff\" (UID: \"1f9f6115-560b-451b-ba0d-e66e67ca55eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0bf34a-754fc9fc-tx5ff" Nov 03 01:30:59.310574 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:59.310549 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f9f6115-560b-451b-ba0d-e66e67ca55eb-metrics-certs\") pod \"maas-api-e2e-worker-w4-0bf34a-754fc9fc-tx5ff\" (UID: \"1f9f6115-560b-451b-ba0d-e66e67ca55eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0bf34a-754fc9fc-tx5ff" Nov 03 01:30:59.371665 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:59.371632 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cngdj\" (UniqueName: \"kubernetes.io/projected/1f9f6115-560b-451b-ba0d-e66e67ca55eb-kube-api-access-cngdj\") pod \"maas-api-e2e-worker-w4-0bf34a-754fc9fc-tx5ff\" (UID: \"1f9f6115-560b-451b-ba0d-e66e67ca55eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0bf34a-754fc9fc-tx5ff" Nov 03 01:30:59.463474 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:59.463354 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:59.669788 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:59.669748 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:59.813156 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:59.813106 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f9f6115-560b-451b-ba0d-e66e67ca55eb-maas-api-tls\") pod \"maas-api-e2e-worker-w4-0bf34a-754fc9fc-tx5ff\" (UID: \"1f9f6115-560b-451b-ba0d-e66e67ca55eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0bf34a-754fc9fc-tx5ff" Nov 03 01:30:59.815747 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:59.815714 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f9f6115-560b-451b-ba0d-e66e67ca55eb-maas-api-tls\") pod \"maas-api-e2e-worker-w4-0bf34a-754fc9fc-tx5ff\" (UID: \"1f9f6115-560b-451b-ba0d-e66e67ca55eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0bf34a-754fc9fc-tx5ff" Nov 03 01:30:59.869631 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:30:59.869592 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:00.090891 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:00.090796 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0bf34a-754fc9fc-tx5ff" Nov 03 01:31:00.461877 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:31:00.461843 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f9f6115_560b_451b_ba0d_e66e67ca55eb.slice/crio-374eeea253bb92017b76b742e39f7b31e4447cbb789266c6f77f43c7158f115e WatchSource:0}: Error finding container 374eeea253bb92017b76b742e39f7b31e4447cbb789266c6f77f43c7158f115e: Status 404 returned error can't find the container with id 374eeea253bb92017b76b742e39f7b31e4447cbb789266c6f77f43c7158f115e Nov 03 01:31:00.463936 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:00.463906 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-0bf34a-754fc9fc-tx5ff"] Nov 03 01:31:00.548234 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:00.548190 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0bf34a-754fc9fc-tx5ff" event={"ID":"1f9f6115-560b-451b-ba0d-e66e67ca55eb","Type":"ContainerStarted","Data":"374eeea253bb92017b76b742e39f7b31e4447cbb789266c6f77f43c7158f115e"} Nov 03 01:31:01.484253 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:01.484226 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0bf34a-66c455cb47-xbxp2" Nov 03 01:31:01.554839 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:01.554790 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0bf34a-754fc9fc-tx5ff" event={"ID":"1f9f6115-560b-451b-ba0d-e66e67ca55eb","Type":"ContainerStarted","Data":"4b7fe5d5bc82cb63ab2aa2a35a6c55ccbe003c78663ae91af039c012c6c1c6cf"} Nov 03 01:31:01.555035 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:01.554860 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0bf34a-754fc9fc-tx5ff" Nov 03 01:31:01.727784 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:01.727661 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0bf34a-754fc9fc-tx5ff" podStartSLOduration=1.768745606 podStartE2EDuration="2.727645501s" totalImagesPullingTime="958.899895ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:30:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:31:01.72671261 +0000 UTC m=+743.756194908" watchObservedRunningTime="2026-11-03 01:31:01.727645501 +0000 UTC m=+743.757127797" Nov 03 01:31:03.505139 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:03.505104 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-b6fb9b-8b66f7f77-5xglx" Nov 03 01:31:03.505737 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:03.505194 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b6fb9b-666f59b59f-7d5tv" Nov 03 01:31:04.528645 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:04.528605 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-b6fb9b-7968d7844-zkbmz"] Nov 03 01:31:04.532716 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:04.532688 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b6fb9b-7968d7844-zkbmz" Nov 03 01:31:04.587480 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:04.587447 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-b6fb9b-7968d7844-zkbmz"] Nov 03 01:31:04.637322 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:04.637288 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-b6fb9b-serving-cert\"" Nov 03 01:31:04.663924 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:04.663880 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/444cb387-71d4-4d99-9e81-603e213995cd-maas-api-tls\") pod \"maas-api-e2e-worker-w0-b6fb9b-7968d7844-zkbmz\" (UID: \"444cb387-71d4-4d99-9e81-603e213995cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b6fb9b-7968d7844-zkbmz" Nov 03 01:31:04.664092 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:04.663933 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5lf9\" (UniqueName: \"kubernetes.io/projected/444cb387-71d4-4d99-9e81-603e213995cd-kube-api-access-d5lf9\") pod \"maas-api-e2e-worker-w0-b6fb9b-7968d7844-zkbmz\" (UID: \"444cb387-71d4-4d99-9e81-603e213995cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b6fb9b-7968d7844-zkbmz" Nov 03 01:31:04.664092 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:04.664053 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/444cb387-71d4-4d99-9e81-603e213995cd-metrics-certs\") pod \"maas-api-e2e-worker-w0-b6fb9b-7968d7844-zkbmz\" (UID: \"444cb387-71d4-4d99-9e81-603e213995cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b6fb9b-7968d7844-zkbmz" Nov 03 01:31:04.664177 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:04.664111 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/444cb387-71d4-4d99-9e81-603e213995cd-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-b6fb9b-7968d7844-zkbmz\" (UID: \"444cb387-71d4-4d99-9e81-603e213995cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b6fb9b-7968d7844-zkbmz" Nov 03 01:31:04.765096 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:04.765051 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/444cb387-71d4-4d99-9e81-603e213995cd-metrics-certs\") pod \"maas-api-e2e-worker-w0-b6fb9b-7968d7844-zkbmz\" (UID: \"444cb387-71d4-4d99-9e81-603e213995cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b6fb9b-7968d7844-zkbmz" Nov 03 01:31:04.765274 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:04.765113 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/444cb387-71d4-4d99-9e81-603e213995cd-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-b6fb9b-7968d7844-zkbmz\" (UID: \"444cb387-71d4-4d99-9e81-603e213995cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b6fb9b-7968d7844-zkbmz" Nov 03 01:31:04.765274 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:04.765238 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/444cb387-71d4-4d99-9e81-603e213995cd-maas-api-tls\") pod \"maas-api-e2e-worker-w0-b6fb9b-7968d7844-zkbmz\" (UID: \"444cb387-71d4-4d99-9e81-603e213995cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b6fb9b-7968d7844-zkbmz" Nov 03 01:31:04.765274 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:04.765265 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5lf9\" (UniqueName: \"kubernetes.io/projected/444cb387-71d4-4d99-9e81-603e213995cd-kube-api-access-d5lf9\") pod \"maas-api-e2e-worker-w0-b6fb9b-7968d7844-zkbmz\" (UID: \"444cb387-71d4-4d99-9e81-603e213995cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b6fb9b-7968d7844-zkbmz" Nov 03 01:31:04.765519 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:31:04.765418 5437 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-b6fb9b-serving-cert: secret "maas-api-e2e-worker-w0-b6fb9b-serving-cert" not found Nov 03 01:31:04.765573 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:31:04.765528 5437 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/444cb387-71d4-4d99-9e81-603e213995cd-maas-api-tls podName:444cb387-71d4-4d99-9e81-603e213995cd nodeName:}" failed. No retries permitted until 2026-11-03 01:31:05.26550124 +0000 UTC m=+747.294983536 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/444cb387-71d4-4d99-9e81-603e213995cd-maas-api-tls") pod "maas-api-e2e-worker-w0-b6fb9b-7968d7844-zkbmz" (UID: "444cb387-71d4-4d99-9e81-603e213995cd") : secret "maas-api-e2e-worker-w0-b6fb9b-serving-cert" not found Nov 03 01:31:04.765870 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:04.765843 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/444cb387-71d4-4d99-9e81-603e213995cd-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-b6fb9b-7968d7844-zkbmz\" (UID: \"444cb387-71d4-4d99-9e81-603e213995cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b6fb9b-7968d7844-zkbmz" Nov 03 01:31:04.767922 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:04.767895 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/444cb387-71d4-4d99-9e81-603e213995cd-metrics-certs\") pod \"maas-api-e2e-worker-w0-b6fb9b-7968d7844-zkbmz\" (UID: \"444cb387-71d4-4d99-9e81-603e213995cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b6fb9b-7968d7844-zkbmz" Nov 03 01:31:04.779608 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:04.779503 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:04.831012 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:04.830977 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5lf9\" (UniqueName: \"kubernetes.io/projected/444cb387-71d4-4d99-9e81-603e213995cd-kube-api-access-d5lf9\") pod \"maas-api-e2e-worker-w0-b6fb9b-7968d7844-zkbmz\" (UID: \"444cb387-71d4-4d99-9e81-603e213995cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b6fb9b-7968d7844-zkbmz" Nov 03 01:31:05.024790 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:05.024756 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:05.229969 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:05.229927 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:05.270298 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:05.270260 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/444cb387-71d4-4d99-9e81-603e213995cd-maas-api-tls\") pod \"maas-api-e2e-worker-w0-b6fb9b-7968d7844-zkbmz\" (UID: \"444cb387-71d4-4d99-9e81-603e213995cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b6fb9b-7968d7844-zkbmz" Nov 03 01:31:05.272930 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:05.272901 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/444cb387-71d4-4d99-9e81-603e213995cd-maas-api-tls\") pod \"maas-api-e2e-worker-w0-b6fb9b-7968d7844-zkbmz\" (UID: \"444cb387-71d4-4d99-9e81-603e213995cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b6fb9b-7968d7844-zkbmz" Nov 03 01:31:05.446147 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:05.446099 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b6fb9b-7968d7844-zkbmz" Nov 03 01:31:05.817871 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:31:05.817838 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod444cb387_71d4_4d99_9e81_603e213995cd.slice/crio-e7dea172b7956882c5a79a9fc01d9ae511fd5bfabdbe8a14931383b212bff40d WatchSource:0}: Error finding container e7dea172b7956882c5a79a9fc01d9ae511fd5bfabdbe8a14931383b212bff40d: Status 404 returned error can't find the container with id e7dea172b7956882c5a79a9fc01d9ae511fd5bfabdbe8a14931383b212bff40d Nov 03 01:31:05.820805 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:05.820779 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-b6fb9b-7968d7844-zkbmz"] Nov 03 01:31:06.586429 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:06.586392 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b6fb9b-7968d7844-zkbmz" event={"ID":"444cb387-71d4-4d99-9e81-603e213995cd","Type":"ContainerStarted","Data":"e7dea172b7956882c5a79a9fc01d9ae511fd5bfabdbe8a14931383b212bff40d"} Nov 03 01:31:07.565656 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:07.565623 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0bf34a-754fc9fc-tx5ff" Nov 03 01:31:07.597642 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:07.597605 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b6fb9b-7968d7844-zkbmz" event={"ID":"444cb387-71d4-4d99-9e81-603e213995cd","Type":"ContainerStarted","Data":"61c18fe1222b3838cdd6704dfc0a2ad2bd6380466d10fb2ec963907add2a0910"} Nov 03 01:31:07.597837 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:07.597738 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b6fb9b-7968d7844-zkbmz" Nov 03 01:31:07.808132 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:07.808076 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b6fb9b-7968d7844-zkbmz" podStartSLOduration=3.003573814 podStartE2EDuration="3.808060837s" totalImagesPullingTime="804.487023ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:31:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:31:07.808012207 +0000 UTC m=+749.837494503" watchObservedRunningTime="2026-11-03 01:31:07.808060837 +0000 UTC m=+749.837543145" Nov 03 01:31:09.311251 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:09.311215 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-21c25193-7c797bdf7-2j4bg"] Nov 03 01:31:09.311703 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:09.311467 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-21c25193-7c797bdf7-2j4bg" podUID="6af58ff8-ed74-48e1-935a-50c22f0e1d3f" containerName="maas-api" containerID="cri-o://3e3eb650aa5b4d8df39ece0bc5fc21c04c489ffcae65a99a421ea466d161df63" gracePeriod=30 Nov 03 01:31:09.464994 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:09.464943 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:09.590184 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:09.590154 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-21c25193-7c797bdf7-2j4bg" Nov 03 01:31:09.610109 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:09.610072 5437 generic.go:356] "Generic (PLEG): container finished" podID="6af58ff8-ed74-48e1-935a-50c22f0e1d3f" containerID="3e3eb650aa5b4d8df39ece0bc5fc21c04c489ffcae65a99a421ea466d161df63" exitCode=0 Nov 03 01:31:09.610316 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:09.610119 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-21c25193-7c797bdf7-2j4bg" Nov 03 01:31:09.610316 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:09.610168 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-21c25193-7c797bdf7-2j4bg" event={"ID":"6af58ff8-ed74-48e1-935a-50c22f0e1d3f","Type":"ContainerDied","Data":"3e3eb650aa5b4d8df39ece0bc5fc21c04c489ffcae65a99a421ea466d161df63"} Nov 03 01:31:09.610316 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:09.610254 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-21c25193-7c797bdf7-2j4bg" event={"ID":"6af58ff8-ed74-48e1-935a-50c22f0e1d3f","Type":"ContainerDied","Data":"57334ed188bde2e4195b73c2e024bcde61d25781caa0ee69beeabe8b32853036"} Nov 03 01:31:09.610316 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:09.610279 5437 scope.go:122] "RemoveContainer" containerID="3e3eb650aa5b4d8df39ece0bc5fc21c04c489ffcae65a99a421ea466d161df63" Nov 03 01:31:09.619991 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:09.619968 5437 scope.go:122] "RemoveContainer" containerID="3e3eb650aa5b4d8df39ece0bc5fc21c04c489ffcae65a99a421ea466d161df63" Nov 03 01:31:09.620424 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:31:09.620384 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e3eb650aa5b4d8df39ece0bc5fc21c04c489ffcae65a99a421ea466d161df63\": container with ID starting with 3e3eb650aa5b4d8df39ece0bc5fc21c04c489ffcae65a99a421ea466d161df63 not found: ID does not exist" containerID="3e3eb650aa5b4d8df39ece0bc5fc21c04c489ffcae65a99a421ea466d161df63" Nov 03 01:31:09.620502 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:09.620446 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e3eb650aa5b4d8df39ece0bc5fc21c04c489ffcae65a99a421ea466d161df63"} err="failed to get container status \"3e3eb650aa5b4d8df39ece0bc5fc21c04c489ffcae65a99a421ea466d161df63\": rpc error: code = NotFound desc = could not find container \"3e3eb650aa5b4d8df39ece0bc5fc21c04c489ffcae65a99a421ea466d161df63\": container with ID starting with 3e3eb650aa5b4d8df39ece0bc5fc21c04c489ffcae65a99a421ea466d161df63 not found: ID does not exist" Nov 03 01:31:09.726238 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:09.726205 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6af58ff8-ed74-48e1-935a-50c22f0e1d3f-kube-api-access-zbkps\" (UniqueName: \"kubernetes.io/projected/6af58ff8-ed74-48e1-935a-50c22f0e1d3f-kube-api-access-zbkps\") pod \"6af58ff8-ed74-48e1-935a-50c22f0e1d3f\" (UID: \"6af58ff8-ed74-48e1-935a-50c22f0e1d3f\") " Nov 03 01:31:09.726485 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:09.726252 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6af58ff8-ed74-48e1-935a-50c22f0e1d3f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6af58ff8-ed74-48e1-935a-50c22f0e1d3f-metrics-certs\") pod \"6af58ff8-ed74-48e1-935a-50c22f0e1d3f\" (UID: \"6af58ff8-ed74-48e1-935a-50c22f0e1d3f\") " Nov 03 01:31:09.726485 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:09.726307 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6af58ff8-ed74-48e1-935a-50c22f0e1d3f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6af58ff8-ed74-48e1-935a-50c22f0e1d3f-openshift-service-ca\") pod \"6af58ff8-ed74-48e1-935a-50c22f0e1d3f\" (UID: \"6af58ff8-ed74-48e1-935a-50c22f0e1d3f\") " Nov 03 01:31:09.726704 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:09.726668 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6af58ff8-ed74-48e1-935a-50c22f0e1d3f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6af58ff8-ed74-48e1-935a-50c22f0e1d3f-maas-api-tls\") pod \"6af58ff8-ed74-48e1-935a-50c22f0e1d3f\" (UID: \"6af58ff8-ed74-48e1-935a-50c22f0e1d3f\") " Nov 03 01:31:09.726876 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:09.726842 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6af58ff8-ed74-48e1-935a-50c22f0e1d3f-openshift-service-ca" pod "6af58ff8-ed74-48e1-935a-50c22f0e1d3f" (UID: "6af58ff8-ed74-48e1-935a-50c22f0e1d3f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:31:09.727158 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:09.727137 5437 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6af58ff8-ed74-48e1-935a-50c22f0e1d3f-openshift-service-ca\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:31:09.729696 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:09.729662 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6af58ff8-ed74-48e1-935a-50c22f0e1d3f-metrics-certs" pod "6af58ff8-ed74-48e1-935a-50c22f0e1d3f" (UID: "6af58ff8-ed74-48e1-935a-50c22f0e1d3f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:31:09.729868 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:09.729825 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6af58ff8-ed74-48e1-935a-50c22f0e1d3f-kube-api-access-zbkps" pod "6af58ff8-ed74-48e1-935a-50c22f0e1d3f" (UID: "6af58ff8-ed74-48e1-935a-50c22f0e1d3f"). InnerVolumeSpecName "kube-api-access-zbkps". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:31:09.730033 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:09.730009 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6af58ff8-ed74-48e1-935a-50c22f0e1d3f-maas-api-tls" pod "6af58ff8-ed74-48e1-935a-50c22f0e1d3f" (UID: "6af58ff8-ed74-48e1-935a-50c22f0e1d3f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:31:09.828596 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:09.828491 5437 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6af58ff8-ed74-48e1-935a-50c22f0e1d3f-metrics-certs\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:31:09.828596 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:09.828534 5437 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6af58ff8-ed74-48e1-935a-50c22f0e1d3f-maas-api-tls\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:31:09.828596 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:09.828550 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbkps\" (UniqueName: \"kubernetes.io/projected/6af58ff8-ed74-48e1-935a-50c22f0e1d3f-kube-api-access-zbkps\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:31:10.038723 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:10.038683 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-21c25193-7c797bdf7-2j4bg"] Nov 03 01:31:10.044467 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:10.044430 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-21c25193-7c797bdf7-2j4bg"] Nov 03 01:31:10.087730 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:10.087651 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6af58ff8-ed74-48e1-935a-50c22f0e1d3f" path="/var/lib/kubelet/pods/6af58ff8-ed74-48e1-935a-50c22f0e1d3f/volumes" Nov 03 01:31:10.682767 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:10.682730 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:11.076093 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:11.076026 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:11.392745 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:11.392654 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:11.736631 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:11.736540 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:12.163865 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:12.163825 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:12.444220 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:12.444112 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:12.664647 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:12.664614 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:12.806239 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:12.806201 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-1e406323-openshift-default-5fd6659f69-rs9dq"] Nov 03 01:31:12.806743 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:12.806428 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-1e406323-openshift-default-5fd6659f69-rs9dq" podUID="576f7646-8212-4ed1-b4f1-2d000cc4072b" containerName="istio-proxy" containerID="cri-o://a3bda85284a34773bbb555652857f2e1ad9c5992c293892ad858589ca28375ba" gracePeriod=30 Nov 03 01:31:12.872856 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:12.872819 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:13.171393 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:13.171261 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:13.489851 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:13.489755 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:13.609217 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:13.609190 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b6fb9b-7968d7844-zkbmz" Nov 03 01:31:13.699065 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:13.699011 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:14.043452 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:31:14.043403 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 01:31:14.424514 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:14.424483 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:14.681443 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:14.681347 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:17.724078 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:17.724036 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:17.930513 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:17.930452 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:18.073149 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:18.073121 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-1e406323-openshift-default-5fd6659f69-rs9dq" Nov 03 01:31:18.132904 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:18.132874 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:18.215537 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:18.215501 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/576f7646-8212-4ed1-b4f1-2d000cc4072b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/576f7646-8212-4ed1-b4f1-2d000cc4072b-istio-envoy\") pod \"576f7646-8212-4ed1-b4f1-2d000cc4072b\" (UID: \"576f7646-8212-4ed1-b4f1-2d000cc4072b\") " Nov 03 01:31:18.215749 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:18.215560 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/576f7646-8212-4ed1-b4f1-2d000cc4072b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/576f7646-8212-4ed1-b4f1-2d000cc4072b-istio-data\") pod \"576f7646-8212-4ed1-b4f1-2d000cc4072b\" (UID: \"576f7646-8212-4ed1-b4f1-2d000cc4072b\") " Nov 03 01:31:18.215749 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:18.215605 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/576f7646-8212-4ed1-b4f1-2d000cc4072b-istio-token\" (UniqueName: \"kubernetes.io/projected/576f7646-8212-4ed1-b4f1-2d000cc4072b-istio-token\") pod \"576f7646-8212-4ed1-b4f1-2d000cc4072b\" (UID: \"576f7646-8212-4ed1-b4f1-2d000cc4072b\") " Nov 03 01:31:18.215749 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:18.215704 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/576f7646-8212-4ed1-b4f1-2d000cc4072b-kube-api-access-l44w6\" (UniqueName: \"kubernetes.io/projected/576f7646-8212-4ed1-b4f1-2d000cc4072b-kube-api-access-l44w6\") pod \"576f7646-8212-4ed1-b4f1-2d000cc4072b\" (UID: \"576f7646-8212-4ed1-b4f1-2d000cc4072b\") " Nov 03 01:31:18.215912 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:18.215887 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/576f7646-8212-4ed1-b4f1-2d000cc4072b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/576f7646-8212-4ed1-b4f1-2d000cc4072b-istiod-ca-cert\") pod \"576f7646-8212-4ed1-b4f1-2d000cc4072b\" (UID: \"576f7646-8212-4ed1-b4f1-2d000cc4072b\") " Nov 03 01:31:18.215964 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:18.215929 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/576f7646-8212-4ed1-b4f1-2d000cc4072b-istio-data" pod "576f7646-8212-4ed1-b4f1-2d000cc4072b" (UID: "576f7646-8212-4ed1-b4f1-2d000cc4072b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:31:18.216027 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:18.215938 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/576f7646-8212-4ed1-b4f1-2d000cc4072b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/576f7646-8212-4ed1-b4f1-2d000cc4072b-istio-podinfo\") pod \"576f7646-8212-4ed1-b4f1-2d000cc4072b\" (UID: \"576f7646-8212-4ed1-b4f1-2d000cc4072b\") " Nov 03 01:31:18.216027 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:18.216005 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/576f7646-8212-4ed1-b4f1-2d000cc4072b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/576f7646-8212-4ed1-b4f1-2d000cc4072b-workload-certs\") pod \"576f7646-8212-4ed1-b4f1-2d000cc4072b\" (UID: \"576f7646-8212-4ed1-b4f1-2d000cc4072b\") " Nov 03 01:31:18.216128 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:18.216059 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/576f7646-8212-4ed1-b4f1-2d000cc4072b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/576f7646-8212-4ed1-b4f1-2d000cc4072b-credential-socket\") pod \"576f7646-8212-4ed1-b4f1-2d000cc4072b\" (UID: \"576f7646-8212-4ed1-b4f1-2d000cc4072b\") " Nov 03 01:31:18.216207 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:18.216140 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/576f7646-8212-4ed1-b4f1-2d000cc4072b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/576f7646-8212-4ed1-b4f1-2d000cc4072b-workload-socket\") pod \"576f7646-8212-4ed1-b4f1-2d000cc4072b\" (UID: \"576f7646-8212-4ed1-b4f1-2d000cc4072b\") " Nov 03 01:31:18.216427 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:18.216400 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/576f7646-8212-4ed1-b4f1-2d000cc4072b-workload-certs" pod "576f7646-8212-4ed1-b4f1-2d000cc4072b" (UID: "576f7646-8212-4ed1-b4f1-2d000cc4072b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:31:18.216532 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:18.216429 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/576f7646-8212-4ed1-b4f1-2d000cc4072b-istiod-ca-cert" pod "576f7646-8212-4ed1-b4f1-2d000cc4072b" (UID: "576f7646-8212-4ed1-b4f1-2d000cc4072b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:31:18.216640 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:18.216618 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/576f7646-8212-4ed1-b4f1-2d000cc4072b-credential-socket" pod "576f7646-8212-4ed1-b4f1-2d000cc4072b" (UID: "576f7646-8212-4ed1-b4f1-2d000cc4072b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:31:18.217072 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:18.216724 5437 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/576f7646-8212-4ed1-b4f1-2d000cc4072b-istio-data\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:31:18.217072 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:18.216744 5437 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/576f7646-8212-4ed1-b4f1-2d000cc4072b-istiod-ca-cert\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:31:18.217072 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:18.216759 5437 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/576f7646-8212-4ed1-b4f1-2d000cc4072b-workload-certs\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:31:18.217072 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:18.216772 5437 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/576f7646-8212-4ed1-b4f1-2d000cc4072b-credential-socket\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:31:18.217072 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:18.216958 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/576f7646-8212-4ed1-b4f1-2d000cc4072b-workload-socket" pod "576f7646-8212-4ed1-b4f1-2d000cc4072b" (UID: "576f7646-8212-4ed1-b4f1-2d000cc4072b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:31:18.218263 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:18.218238 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/576f7646-8212-4ed1-b4f1-2d000cc4072b-kube-api-access-l44w6" pod "576f7646-8212-4ed1-b4f1-2d000cc4072b" (UID: "576f7646-8212-4ed1-b4f1-2d000cc4072b"). InnerVolumeSpecName "kube-api-access-l44w6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:31:18.218484 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:18.218464 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/576f7646-8212-4ed1-b4f1-2d000cc4072b-istio-podinfo" pod "576f7646-8212-4ed1-b4f1-2d000cc4072b" (UID: "576f7646-8212-4ed1-b4f1-2d000cc4072b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:31:18.218868 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:18.218850 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/576f7646-8212-4ed1-b4f1-2d000cc4072b-istio-envoy" pod "576f7646-8212-4ed1-b4f1-2d000cc4072b" (UID: "576f7646-8212-4ed1-b4f1-2d000cc4072b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:31:18.218943 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:18.218873 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/576f7646-8212-4ed1-b4f1-2d000cc4072b-istio-token" pod "576f7646-8212-4ed1-b4f1-2d000cc4072b" (UID: "576f7646-8212-4ed1-b4f1-2d000cc4072b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:31:18.318255 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:18.318214 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l44w6\" (UniqueName: \"kubernetes.io/projected/576f7646-8212-4ed1-b4f1-2d000cc4072b-kube-api-access-l44w6\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:31:18.318255 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:18.318253 5437 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/576f7646-8212-4ed1-b4f1-2d000cc4072b-istio-podinfo\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:31:18.318255 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:18.318269 5437 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/576f7646-8212-4ed1-b4f1-2d000cc4072b-workload-socket\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:31:18.318566 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:18.318282 5437 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/576f7646-8212-4ed1-b4f1-2d000cc4072b-istio-envoy\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:31:18.318566 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:18.318294 5437 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/576f7646-8212-4ed1-b4f1-2d000cc4072b-istio-token\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:31:18.661576 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:18.661494 5437 generic.go:356] "Generic (PLEG): container finished" podID="576f7646-8212-4ed1-b4f1-2d000cc4072b" containerID="a3bda85284a34773bbb555652857f2e1ad9c5992c293892ad858589ca28375ba" exitCode=0 Nov 03 01:31:18.661576 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:18.661549 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-1e406323-openshift-default-5fd6659f69-rs9dq" Nov 03 01:31:18.661758 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:18.661582 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-1e406323-openshift-default-5fd6659f69-rs9dq" event={"ID":"576f7646-8212-4ed1-b4f1-2d000cc4072b","Type":"ContainerDied","Data":"a3bda85284a34773bbb555652857f2e1ad9c5992c293892ad858589ca28375ba"} Nov 03 01:31:18.661758 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:18.661642 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-1e406323-openshift-default-5fd6659f69-rs9dq" event={"ID":"576f7646-8212-4ed1-b4f1-2d000cc4072b","Type":"ContainerDied","Data":"b889b0cf5d7d18519630d41d9b6befec7bd9798e04c06e35a4a5798afa06195a"} Nov 03 01:31:18.661758 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:18.661676 5437 scope.go:122] "RemoveContainer" containerID="a3bda85284a34773bbb555652857f2e1ad9c5992c293892ad858589ca28375ba" Nov 03 01:31:18.676614 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:18.672666 5437 scope.go:122] "RemoveContainer" containerID="07efcd1440f2d9ab1a3fd5dfa898f5f3fb995fdce8e0091bc90a4dcce598b4e0" Nov 03 01:31:18.684287 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:18.684260 5437 scope.go:122] "RemoveContainer" containerID="a3bda85284a34773bbb555652857f2e1ad9c5992c293892ad858589ca28375ba" Nov 03 01:31:18.684677 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:31:18.684651 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3bda85284a34773bbb555652857f2e1ad9c5992c293892ad858589ca28375ba\": container with ID starting with a3bda85284a34773bbb555652857f2e1ad9c5992c293892ad858589ca28375ba not found: ID does not exist" containerID="a3bda85284a34773bbb555652857f2e1ad9c5992c293892ad858589ca28375ba" Nov 03 01:31:18.684773 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:18.684692 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3bda85284a34773bbb555652857f2e1ad9c5992c293892ad858589ca28375ba"} err="failed to get container status \"a3bda85284a34773bbb555652857f2e1ad9c5992c293892ad858589ca28375ba\": rpc error: code = NotFound desc = could not find container \"a3bda85284a34773bbb555652857f2e1ad9c5992c293892ad858589ca28375ba\": container with ID starting with a3bda85284a34773bbb555652857f2e1ad9c5992c293892ad858589ca28375ba not found: ID does not exist" Nov 03 01:31:18.684773 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:18.684720 5437 scope.go:122] "RemoveContainer" containerID="07efcd1440f2d9ab1a3fd5dfa898f5f3fb995fdce8e0091bc90a4dcce598b4e0" Nov 03 01:31:18.685047 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:31:18.685027 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07efcd1440f2d9ab1a3fd5dfa898f5f3fb995fdce8e0091bc90a4dcce598b4e0\": container with ID starting with 07efcd1440f2d9ab1a3fd5dfa898f5f3fb995fdce8e0091bc90a4dcce598b4e0 not found: ID does not exist" containerID="07efcd1440f2d9ab1a3fd5dfa898f5f3fb995fdce8e0091bc90a4dcce598b4e0" Nov 03 01:31:18.685085 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:18.685054 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07efcd1440f2d9ab1a3fd5dfa898f5f3fb995fdce8e0091bc90a4dcce598b4e0"} err="failed to get container status \"07efcd1440f2d9ab1a3fd5dfa898f5f3fb995fdce8e0091bc90a4dcce598b4e0\": rpc error: code = NotFound desc = could not find container \"07efcd1440f2d9ab1a3fd5dfa898f5f3fb995fdce8e0091bc90a4dcce598b4e0\": container with ID starting with 07efcd1440f2d9ab1a3fd5dfa898f5f3fb995fdce8e0091bc90a4dcce598b4e0 not found: ID does not exist" Nov 03 01:31:18.700313 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:18.700277 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:18.832318 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:18.832272 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-1e406323-openshift-default-5fd6659f69-rs9dq"] Nov 03 01:31:18.883626 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:18.883590 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-1e406323-openshift-default-5fd6659f69-rs9dq"] Nov 03 01:31:18.951753 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:18.951651 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:20.087276 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:20.087237 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="576f7646-8212-4ed1-b4f1-2d000cc4072b" path="/var/lib/kubelet/pods/576f7646-8212-4ed1-b4f1-2d000cc4072b/volumes" Nov 03 01:31:20.946963 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:20.946919 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:21.143924 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:21.143866 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:21.289202 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:21.289108 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-9da431-69447f989f-7qws7"] Nov 03 01:31:21.289624 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:21.289609 5437 cpu_manager.go:414] "containerMap: removing container" podUID="576f7646-8212-4ed1-b4f1-2d000cc4072b" containerName="istio-proxy" Nov 03 01:31:21.289681 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:21.289625 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="576f7646-8212-4ed1-b4f1-2d000cc4072b" containerName="istio-proxy" Nov 03 01:31:21.289681 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:21.289636 5437 cpu_manager.go:414] "containerMap: removing container" podUID="6af58ff8-ed74-48e1-935a-50c22f0e1d3f" containerName="maas-api" Nov 03 01:31:21.289681 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:21.289642 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6af58ff8-ed74-48e1-935a-50c22f0e1d3f" containerName="maas-api" Nov 03 01:31:21.289681 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:21.289663 5437 cpu_manager.go:414] "containerMap: removing container" podUID="576f7646-8212-4ed1-b4f1-2d000cc4072b" containerName="istio-proxy" Nov 03 01:31:21.289681 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:21.289669 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="576f7646-8212-4ed1-b4f1-2d000cc4072b" containerName="istio-proxy" Nov 03 01:31:21.289854 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:21.289730 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="576f7646-8212-4ed1-b4f1-2d000cc4072b" containerName="istio-proxy" Nov 03 01:31:21.289854 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:21.289741 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="6af58ff8-ed74-48e1-935a-50c22f0e1d3f" containerName="maas-api" Nov 03 01:31:21.292721 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:21.292702 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-9da431-69447f989f-7qws7" Nov 03 01:31:21.346455 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:21.346411 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-9da431-69447f989f-7qws7"] Nov 03 01:31:21.396693 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:21.396658 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-9da431\"" Nov 03 01:31:21.396872 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:21.396715 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-9da431-dockercfg-g9fzj\"" Nov 03 01:31:21.450129 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:21.450091 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-9da431-756c59d995-5p6n9"] Nov 03 01:31:21.450665 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:21.450650 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="576f7646-8212-4ed1-b4f1-2d000cc4072b" containerName="istio-proxy" Nov 03 01:31:21.451985 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:21.451957 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxvbw\" (UniqueName: \"kubernetes.io/projected/952ba771-48ba-43ed-8ceb-290a0e79ebdb-kube-api-access-zxvbw\") pod \"payload-pre-processing-e2e-worker-w2-9da431-69447f989f-7qws7\" (UID: \"952ba771-48ba-43ed-8ceb-290a0e79ebdb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-9da431-69447f989f-7qws7" Nov 03 01:31:21.452145 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:21.452064 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/952ba771-48ba-43ed-8ceb-290a0e79ebdb-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-9da431-69447f989f-7qws7\" (UID: \"952ba771-48ba-43ed-8ceb-290a0e79ebdb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-9da431-69447f989f-7qws7" Nov 03 01:31:21.453610 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:21.453592 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-9da431-756c59d995-5p6n9" Nov 03 01:31:21.507639 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:21.507605 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-9da431-756c59d995-5p6n9"] Nov 03 01:31:21.552727 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:21.552691 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxvbw\" (UniqueName: \"kubernetes.io/projected/952ba771-48ba-43ed-8ceb-290a0e79ebdb-kube-api-access-zxvbw\") pod \"payload-pre-processing-e2e-worker-w2-9da431-69447f989f-7qws7\" (UID: \"952ba771-48ba-43ed-8ceb-290a0e79ebdb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-9da431-69447f989f-7qws7" Nov 03 01:31:21.552938 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:21.552746 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/952ba771-48ba-43ed-8ceb-290a0e79ebdb-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-9da431-69447f989f-7qws7\" (UID: \"952ba771-48ba-43ed-8ceb-290a0e79ebdb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-9da431-69447f989f-7qws7" Nov 03 01:31:21.552938 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:21.552772 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4df6\" (UniqueName: \"kubernetes.io/projected/a7e0c5cd-7758-4a54-9a9f-18797ffe2568-kube-api-access-x4df6\") pod \"payload-processing-e2e-worker-w2-9da431-756c59d995-5p6n9\" (UID: \"a7e0c5cd-7758-4a54-9a9f-18797ffe2568\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-9da431-756c59d995-5p6n9" Nov 03 01:31:21.552938 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:21.552858 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7e0c5cd-7758-4a54-9a9f-18797ffe2568-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-9da431-756c59d995-5p6n9\" (UID: \"a7e0c5cd-7758-4a54-9a9f-18797ffe2568\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-9da431-756c59d995-5p6n9" Nov 03 01:31:21.553556 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:21.553535 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/952ba771-48ba-43ed-8ceb-290a0e79ebdb-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-9da431-69447f989f-7qws7\" (UID: \"952ba771-48ba-43ed-8ceb-290a0e79ebdb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-9da431-69447f989f-7qws7" Nov 03 01:31:21.616068 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:21.616033 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxvbw\" (UniqueName: \"kubernetes.io/projected/952ba771-48ba-43ed-8ceb-290a0e79ebdb-kube-api-access-zxvbw\") pod \"payload-pre-processing-e2e-worker-w2-9da431-69447f989f-7qws7\" (UID: \"952ba771-48ba-43ed-8ceb-290a0e79ebdb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-9da431-69447f989f-7qws7" Nov 03 01:31:21.653941 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:21.653896 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4df6\" (UniqueName: \"kubernetes.io/projected/a7e0c5cd-7758-4a54-9a9f-18797ffe2568-kube-api-access-x4df6\") pod \"payload-processing-e2e-worker-w2-9da431-756c59d995-5p6n9\" (UID: \"a7e0c5cd-7758-4a54-9a9f-18797ffe2568\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-9da431-756c59d995-5p6n9" Nov 03 01:31:21.654139 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:21.654011 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7e0c5cd-7758-4a54-9a9f-18797ffe2568-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-9da431-756c59d995-5p6n9\" (UID: \"a7e0c5cd-7758-4a54-9a9f-18797ffe2568\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-9da431-756c59d995-5p6n9" Nov 03 01:31:21.654770 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:21.654747 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7e0c5cd-7758-4a54-9a9f-18797ffe2568-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-9da431-756c59d995-5p6n9\" (UID: \"a7e0c5cd-7758-4a54-9a9f-18797ffe2568\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-9da431-756c59d995-5p6n9" Nov 03 01:31:21.716430 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:21.716395 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4df6\" (UniqueName: \"kubernetes.io/projected/a7e0c5cd-7758-4a54-9a9f-18797ffe2568-kube-api-access-x4df6\") pod \"payload-processing-e2e-worker-w2-9da431-756c59d995-5p6n9\" (UID: \"a7e0c5cd-7758-4a54-9a9f-18797ffe2568\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-9da431-756c59d995-5p6n9" Nov 03 01:31:21.769490 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:21.769450 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-9da431-756c59d995-5p6n9" Nov 03 01:31:21.905608 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:21.905553 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-9da431-69447f989f-7qws7" Nov 03 01:31:22.167144 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:22.165929 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-9da431-756c59d995-5p6n9"] Nov 03 01:31:22.286207 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:31:22.286154 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod952ba771_48ba_43ed_8ceb_290a0e79ebdb.slice/crio-137edb5df23072a637f814d4bac4ec674254fd828bafd08687bd7d678112d006 WatchSource:0}: Error finding container 137edb5df23072a637f814d4bac4ec674254fd828bafd08687bd7d678112d006: Status 404 returned error can't find the container with id 137edb5df23072a637f814d4bac4ec674254fd828bafd08687bd7d678112d006 Nov 03 01:31:22.288332 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:22.288304 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-9da431-69447f989f-7qws7"] Nov 03 01:31:22.685051 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:22.685008 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-9da431-756c59d995-5p6n9" event={"ID":"a7e0c5cd-7758-4a54-9a9f-18797ffe2568","Type":"ContainerStarted","Data":"630e37c39b6a34b7dd731aa34f7b824396861196cfef1f3328bac832b4f1121a"} Nov 03 01:31:22.685290 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:22.685083 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-9da431-756c59d995-5p6n9" event={"ID":"a7e0c5cd-7758-4a54-9a9f-18797ffe2568","Type":"ContainerStarted","Data":"ef810443ae0061bd04f3f645c99bc1f634d8404c003612cbb52dd1af700dc233"} Nov 03 01:31:22.685290 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:22.685116 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-9da431-756c59d995-5p6n9" Nov 03 01:31:22.686744 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:22.686713 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-9da431-69447f989f-7qws7" event={"ID":"952ba771-48ba-43ed-8ceb-290a0e79ebdb","Type":"ContainerStarted","Data":"22f5233c31242f73f0b78b19566ccc8d3cc80c7d56f265d0dd229484f2d1c3bc"} Nov 03 01:31:22.686871 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:22.686767 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-9da431-69447f989f-7qws7" event={"ID":"952ba771-48ba-43ed-8ceb-290a0e79ebdb","Type":"ContainerStarted","Data":"137edb5df23072a637f814d4bac4ec674254fd828bafd08687bd7d678112d006"} Nov 03 01:31:22.686871 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:22.686804 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-9da431-69447f989f-7qws7" Nov 03 01:31:22.804966 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:22.804911 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-9da431-756c59d995-5p6n9" podStartSLOduration=1.804896404 podStartE2EDuration="1.804896404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:31:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:31:22.804135969 +0000 UTC m=+764.833618265" watchObservedRunningTime="2026-11-03 01:31:22.804896404 +0000 UTC m=+764.834378698" Nov 03 01:31:22.924294 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:22.924240 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-9da431-69447f989f-7qws7" podStartSLOduration=1.9242280250000001 podStartE2EDuration="1.924228025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:31:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:31:22.923522669 +0000 UTC m=+764.953004964" watchObservedRunningTime="2026-11-03 01:31:22.924228025 +0000 UTC m=+764.953710319" Nov 03 01:31:23.113106 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:23.113066 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:23.998770 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:23.998735 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:24.599106 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:24.599067 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:24.848493 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:24.848457 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:27.816010 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:27.815971 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:28.028046 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:28.028001 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:28.170551 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:28.170463 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-e1793d-7f578596cf-fj8br"] Nov 03 01:31:28.175457 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:28.175426 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e1793d-7f578596cf-fj8br" Nov 03 01:31:28.217488 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:28.217448 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3029fee9-a1c7-4340-b52f-5fe182e44252-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-e1793d-7f578596cf-fj8br\" (UID: \"3029fee9-a1c7-4340-b52f-5fe182e44252\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e1793d-7f578596cf-fj8br" Nov 03 01:31:28.217706 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:28.217589 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjlwd\" (UniqueName: \"kubernetes.io/projected/3029fee9-a1c7-4340-b52f-5fe182e44252-kube-api-access-jjlwd\") pod \"payload-pre-processing-e2e-worker-w1-e1793d-7f578596cf-fj8br\" (UID: \"3029fee9-a1c7-4340-b52f-5fe182e44252\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e1793d-7f578596cf-fj8br" Nov 03 01:31:28.229855 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:28.229774 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-e1793d\"" Nov 03 01:31:28.230084 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:28.229880 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-e1793d-7f578596cf-fj8br"] Nov 03 01:31:28.230431 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:28.230407 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-e1793d-dockercfg-xp6rq\"" Nov 03 01:31:28.237260 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:28.237202 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:28.318471 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:28.318433 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3029fee9-a1c7-4340-b52f-5fe182e44252-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-e1793d-7f578596cf-fj8br\" (UID: \"3029fee9-a1c7-4340-b52f-5fe182e44252\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e1793d-7f578596cf-fj8br" Nov 03 01:31:28.318631 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:28.318486 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjlwd\" (UniqueName: \"kubernetes.io/projected/3029fee9-a1c7-4340-b52f-5fe182e44252-kube-api-access-jjlwd\") pod \"payload-pre-processing-e2e-worker-w1-e1793d-7f578596cf-fj8br\" (UID: \"3029fee9-a1c7-4340-b52f-5fe182e44252\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e1793d-7f578596cf-fj8br" Nov 03 01:31:28.319461 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:28.319434 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3029fee9-a1c7-4340-b52f-5fe182e44252-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-e1793d-7f578596cf-fj8br\" (UID: \"3029fee9-a1c7-4340-b52f-5fe182e44252\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e1793d-7f578596cf-fj8br" Nov 03 01:31:28.381814 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:28.381778 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjlwd\" (UniqueName: \"kubernetes.io/projected/3029fee9-a1c7-4340-b52f-5fe182e44252-kube-api-access-jjlwd\") pod \"payload-pre-processing-e2e-worker-w1-e1793d-7f578596cf-fj8br\" (UID: \"3029fee9-a1c7-4340-b52f-5fe182e44252\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e1793d-7f578596cf-fj8br" Nov 03 01:31:28.489739 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:28.489639 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e1793d-7f578596cf-fj8br" Nov 03 01:31:28.866354 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:31:28.866319 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3029fee9_a1c7_4340_b52f_5fe182e44252.slice/crio-9be001cc7cee69c1b7ac35973a1711158cf505f7dda11bc41f9e8e416d1509da WatchSource:0}: Error finding container 9be001cc7cee69c1b7ac35973a1711158cf505f7dda11bc41f9e8e416d1509da: Status 404 returned error can't find the container with id 9be001cc7cee69c1b7ac35973a1711158cf505f7dda11bc41f9e8e416d1509da Nov 03 01:31:28.868548 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:28.868525 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-e1793d-7f578596cf-fj8br"] Nov 03 01:31:29.351471 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:29.351436 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:29.604015 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:29.603906 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:29.731477 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:29.731440 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e1793d-7f578596cf-fj8br" event={"ID":"3029fee9-a1c7-4340-b52f-5fe182e44252","Type":"ContainerStarted","Data":"48a451ee9203ad15ae0cebeed7d9004998590005542499f27a85333f30a3bc52"} Nov 03 01:31:29.731672 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:29.731497 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e1793d-7f578596cf-fj8br" event={"ID":"3029fee9-a1c7-4340-b52f-5fe182e44252","Type":"ContainerStarted","Data":"9be001cc7cee69c1b7ac35973a1711158cf505f7dda11bc41f9e8e416d1509da"} Nov 03 01:31:29.731672 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:29.731644 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e1793d-7f578596cf-fj8br" Nov 03 01:31:29.746667 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:29.746633 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-41aa5f/simulator-w3-41aa5f-kserve-586cfbcdd4-gsmfk"] Nov 03 01:31:29.750354 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:29.750324 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-41aa5f/simulator-w3-41aa5f-kserve-586cfbcdd4-gsmfk" Nov 03 01:31:29.805112 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:29.805073 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-41aa5f/simulator-w3-41aa5f-kserve-586cfbcdd4-gsmfk"] Nov 03 01:31:29.837232 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:29.837191 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a87f809-0b26-480e-bb37-ff234a40c5c5-model-cache\") pod \"simulator-w3-41aa5f-kserve-586cfbcdd4-gsmfk\" (UID: \"8a87f809-0b26-480e-bb37-ff234a40c5c5\") " pod="e2e-models-e2e-worker-w3-41aa5f/simulator-w3-41aa5f-kserve-586cfbcdd4-gsmfk" Nov 03 01:31:29.837485 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:29.837387 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a87f809-0b26-480e-bb37-ff234a40c5c5-tls-certs\") pod \"simulator-w3-41aa5f-kserve-586cfbcdd4-gsmfk\" (UID: \"8a87f809-0b26-480e-bb37-ff234a40c5c5\") " pod="e2e-models-e2e-worker-w3-41aa5f/simulator-w3-41aa5f-kserve-586cfbcdd4-gsmfk" Nov 03 01:31:29.837485 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:29.837436 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwqzh\" (UniqueName: \"kubernetes.io/projected/8a87f809-0b26-480e-bb37-ff234a40c5c5-kube-api-access-cwqzh\") pod \"simulator-w3-41aa5f-kserve-586cfbcdd4-gsmfk\" (UID: \"8a87f809-0b26-480e-bb37-ff234a40c5c5\") " pod="e2e-models-e2e-worker-w3-41aa5f/simulator-w3-41aa5f-kserve-586cfbcdd4-gsmfk" Nov 03 01:31:29.837668 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:29.837533 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a87f809-0b26-480e-bb37-ff234a40c5c5-tmp-dir\") pod \"simulator-w3-41aa5f-kserve-586cfbcdd4-gsmfk\" (UID: \"8a87f809-0b26-480e-bb37-ff234a40c5c5\") " pod="e2e-models-e2e-worker-w3-41aa5f/simulator-w3-41aa5f-kserve-586cfbcdd4-gsmfk" Nov 03 01:31:29.837668 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:29.837564 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a87f809-0b26-480e-bb37-ff234a40c5c5-dshm\") pod \"simulator-w3-41aa5f-kserve-586cfbcdd4-gsmfk\" (UID: \"8a87f809-0b26-480e-bb37-ff234a40c5c5\") " pod="e2e-models-e2e-worker-w3-41aa5f/simulator-w3-41aa5f-kserve-586cfbcdd4-gsmfk" Nov 03 01:31:29.837668 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:29.837602 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a87f809-0b26-480e-bb37-ff234a40c5c5-home\") pod \"simulator-w3-41aa5f-kserve-586cfbcdd4-gsmfk\" (UID: \"8a87f809-0b26-480e-bb37-ff234a40c5c5\") " pod="e2e-models-e2e-worker-w3-41aa5f/simulator-w3-41aa5f-kserve-586cfbcdd4-gsmfk" Nov 03 01:31:29.838054 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:29.838037 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-41aa5f\"/\"kube-root-ca.crt\"" Nov 03 01:31:29.838112 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:29.838076 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-41aa5f\"/\"openshift-service-ca.crt\"" Nov 03 01:31:29.848534 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:29.848498 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-41aa5f\"/\"default-dockercfg-c22xk\"" Nov 03 01:31:29.848534 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:29.848498 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-41aa5f\"/\"simulator-w3-41aa5f-kserve-self-signed-certs\"" Nov 03 01:31:29.889733 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:29.889625 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e1793d-7f578596cf-fj8br" podStartSLOduration=1.8896115999999998 podStartE2EDuration="1.8896116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:31:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:31:29.889281187 +0000 UTC m=+771.918763482" watchObservedRunningTime="2026-11-03 01:31:29.8896116 +0000 UTC m=+771.919093895" Nov 03 01:31:29.891598 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:29.891568 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:29.938406 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:29.938351 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a87f809-0b26-480e-bb37-ff234a40c5c5-model-cache\") pod \"simulator-w3-41aa5f-kserve-586cfbcdd4-gsmfk\" (UID: \"8a87f809-0b26-480e-bb37-ff234a40c5c5\") " pod="e2e-models-e2e-worker-w3-41aa5f/simulator-w3-41aa5f-kserve-586cfbcdd4-gsmfk" Nov 03 01:31:29.938611 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:29.938423 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a87f809-0b26-480e-bb37-ff234a40c5c5-tls-certs\") pod \"simulator-w3-41aa5f-kserve-586cfbcdd4-gsmfk\" (UID: \"8a87f809-0b26-480e-bb37-ff234a40c5c5\") " pod="e2e-models-e2e-worker-w3-41aa5f/simulator-w3-41aa5f-kserve-586cfbcdd4-gsmfk" Nov 03 01:31:29.938611 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:29.938447 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwqzh\" (UniqueName: \"kubernetes.io/projected/8a87f809-0b26-480e-bb37-ff234a40c5c5-kube-api-access-cwqzh\") pod \"simulator-w3-41aa5f-kserve-586cfbcdd4-gsmfk\" (UID: \"8a87f809-0b26-480e-bb37-ff234a40c5c5\") " pod="e2e-models-e2e-worker-w3-41aa5f/simulator-w3-41aa5f-kserve-586cfbcdd4-gsmfk" Nov 03 01:31:29.938611 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:29.938488 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a87f809-0b26-480e-bb37-ff234a40c5c5-tmp-dir\") pod \"simulator-w3-41aa5f-kserve-586cfbcdd4-gsmfk\" (UID: \"8a87f809-0b26-480e-bb37-ff234a40c5c5\") " pod="e2e-models-e2e-worker-w3-41aa5f/simulator-w3-41aa5f-kserve-586cfbcdd4-gsmfk" Nov 03 01:31:29.938611 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:29.938517 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a87f809-0b26-480e-bb37-ff234a40c5c5-dshm\") pod \"simulator-w3-41aa5f-kserve-586cfbcdd4-gsmfk\" (UID: \"8a87f809-0b26-480e-bb37-ff234a40c5c5\") " pod="e2e-models-e2e-worker-w3-41aa5f/simulator-w3-41aa5f-kserve-586cfbcdd4-gsmfk" Nov 03 01:31:29.938611 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:29.938553 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a87f809-0b26-480e-bb37-ff234a40c5c5-home\") pod \"simulator-w3-41aa5f-kserve-586cfbcdd4-gsmfk\" (UID: \"8a87f809-0b26-480e-bb37-ff234a40c5c5\") " pod="e2e-models-e2e-worker-w3-41aa5f/simulator-w3-41aa5f-kserve-586cfbcdd4-gsmfk" Nov 03 01:31:29.938866 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:29.938845 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a87f809-0b26-480e-bb37-ff234a40c5c5-model-cache\") pod \"simulator-w3-41aa5f-kserve-586cfbcdd4-gsmfk\" (UID: \"8a87f809-0b26-480e-bb37-ff234a40c5c5\") " pod="e2e-models-e2e-worker-w3-41aa5f/simulator-w3-41aa5f-kserve-586cfbcdd4-gsmfk" Nov 03 01:31:29.938931 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:29.938910 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a87f809-0b26-480e-bb37-ff234a40c5c5-tmp-dir\") pod \"simulator-w3-41aa5f-kserve-586cfbcdd4-gsmfk\" (UID: \"8a87f809-0b26-480e-bb37-ff234a40c5c5\") " pod="e2e-models-e2e-worker-w3-41aa5f/simulator-w3-41aa5f-kserve-586cfbcdd4-gsmfk" Nov 03 01:31:29.938990 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:29.938977 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a87f809-0b26-480e-bb37-ff234a40c5c5-home\") pod \"simulator-w3-41aa5f-kserve-586cfbcdd4-gsmfk\" (UID: \"8a87f809-0b26-480e-bb37-ff234a40c5c5\") " pod="e2e-models-e2e-worker-w3-41aa5f/simulator-w3-41aa5f-kserve-586cfbcdd4-gsmfk" Nov 03 01:31:29.940933 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:29.940898 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a87f809-0b26-480e-bb37-ff234a40c5c5-dshm\") pod \"simulator-w3-41aa5f-kserve-586cfbcdd4-gsmfk\" (UID: \"8a87f809-0b26-480e-bb37-ff234a40c5c5\") " pod="e2e-models-e2e-worker-w3-41aa5f/simulator-w3-41aa5f-kserve-586cfbcdd4-gsmfk" Nov 03 01:31:29.941123 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:29.941108 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a87f809-0b26-480e-bb37-ff234a40c5c5-tls-certs\") pod \"simulator-w3-41aa5f-kserve-586cfbcdd4-gsmfk\" (UID: \"8a87f809-0b26-480e-bb37-ff234a40c5c5\") " pod="e2e-models-e2e-worker-w3-41aa5f/simulator-w3-41aa5f-kserve-586cfbcdd4-gsmfk" Nov 03 01:31:30.002093 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:30.002054 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwqzh\" (UniqueName: \"kubernetes.io/projected/8a87f809-0b26-480e-bb37-ff234a40c5c5-kube-api-access-cwqzh\") pod \"simulator-w3-41aa5f-kserve-586cfbcdd4-gsmfk\" (UID: \"8a87f809-0b26-480e-bb37-ff234a40c5c5\") " pod="e2e-models-e2e-worker-w3-41aa5f/simulator-w3-41aa5f-kserve-586cfbcdd4-gsmfk" Nov 03 01:31:30.064077 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:30.064020 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-41aa5f/simulator-w3-41aa5f-kserve-586cfbcdd4-gsmfk" Nov 03 01:31:30.441107 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:31:30.441066 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8a87f809_0b26_480e_bb37_ff234a40c5c5.slice/crio-5b1401233b90161766d39ae0c59c80ee2934ae2bf40ceab42423f60dc3a0d8bd WatchSource:0}: Error finding container 5b1401233b90161766d39ae0c59c80ee2934ae2bf40ceab42423f60dc3a0d8bd: Status 404 returned error can't find the container with id 5b1401233b90161766d39ae0c59c80ee2934ae2bf40ceab42423f60dc3a0d8bd Nov 03 01:31:30.443347 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:30.443305 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-41aa5f/simulator-w3-41aa5f-kserve-586cfbcdd4-gsmfk"] Nov 03 01:31:30.738546 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:30.738439 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-41aa5f/simulator-w3-41aa5f-kserve-586cfbcdd4-gsmfk" event={"ID":"8a87f809-0b26-480e-bb37-ff234a40c5c5","Type":"ContainerStarted","Data":"ff1fece57a2d906fc223c5efd375595ffd5ec9aff94298218b650bd18d6aafcb"} Nov 03 01:31:30.738546 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:30.738497 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-41aa5f/simulator-w3-41aa5f-kserve-586cfbcdd4-gsmfk" event={"ID":"8a87f809-0b26-480e-bb37-ff234a40c5c5","Type":"ContainerStarted","Data":"5b1401233b90161766d39ae0c59c80ee2934ae2bf40ceab42423f60dc3a0d8bd"} Nov 03 01:31:30.859461 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:30.859404 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-41aa5f/simulator-w3-41aa5f-kserve-586cfbcdd4-gsmfk" podStartSLOduration=1.8593878369999999 podStartE2EDuration="1.859387837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:31:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:31:30.858909972 +0000 UTC m=+772.888392267" watchObservedRunningTime="2026-11-03 01:31:30.859387837 +0000 UTC m=+772.888870124" Nov 03 01:31:31.064833 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:31.064780 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-41aa5f/simulator-w3-41aa5f-kserve-586cfbcdd4-gsmfk" Nov 03 01:31:33.370297 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:33.370248 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:33.695613 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:33.695538 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-9da431-69447f989f-7qws7" Nov 03 01:31:33.695613 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:33.695600 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-9da431-756c59d995-5p6n9" Nov 03 01:31:34.366090 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:34.366049 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:34.993542 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:34.993498 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:35.247896 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:35.247798 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:36.307772 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:36.307737 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:37.013540 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:37.013502 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:37.214214 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:37.214174 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:38.521327 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:38.521232 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-9da431-79b57b969c-4n9ts"] Nov 03 01:31:38.527295 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:38.527266 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9da431-79b57b969c-4n9ts" Nov 03 01:31:38.579653 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:38.579617 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-9da431-79b57b969c-4n9ts"] Nov 03 01:31:38.627217 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:38.627175 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f277337a-c920-4c20-a029-215d4ee996f9-maas-api-tls\") pod \"maas-api-e2e-worker-w2-9da431-79b57b969c-4n9ts\" (UID: \"f277337a-c920-4c20-a029-215d4ee996f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9da431-79b57b969c-4n9ts" Nov 03 01:31:38.627450 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:38.627233 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f277337a-c920-4c20-a029-215d4ee996f9-metrics-certs\") pod \"maas-api-e2e-worker-w2-9da431-79b57b969c-4n9ts\" (UID: \"f277337a-c920-4c20-a029-215d4ee996f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9da431-79b57b969c-4n9ts" Nov 03 01:31:38.627450 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:38.627353 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g682j\" (UniqueName: \"kubernetes.io/projected/f277337a-c920-4c20-a029-215d4ee996f9-kube-api-access-g682j\") pod \"maas-api-e2e-worker-w2-9da431-79b57b969c-4n9ts\" (UID: \"f277337a-c920-4c20-a029-215d4ee996f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9da431-79b57b969c-4n9ts" Nov 03 01:31:38.627544 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:38.627504 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f277337a-c920-4c20-a029-215d4ee996f9-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-9da431-79b57b969c-4n9ts\" (UID: \"f277337a-c920-4c20-a029-215d4ee996f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9da431-79b57b969c-4n9ts" Nov 03 01:31:38.629334 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:38.629306 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-9da431-serving-cert\"" Nov 03 01:31:38.728672 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:38.728625 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g682j\" (UniqueName: \"kubernetes.io/projected/f277337a-c920-4c20-a029-215d4ee996f9-kube-api-access-g682j\") pod \"maas-api-e2e-worker-w2-9da431-79b57b969c-4n9ts\" (UID: \"f277337a-c920-4c20-a029-215d4ee996f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9da431-79b57b969c-4n9ts" Nov 03 01:31:38.728878 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:38.728759 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f277337a-c920-4c20-a029-215d4ee996f9-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-9da431-79b57b969c-4n9ts\" (UID: \"f277337a-c920-4c20-a029-215d4ee996f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9da431-79b57b969c-4n9ts" Nov 03 01:31:38.728878 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:38.728806 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f277337a-c920-4c20-a029-215d4ee996f9-maas-api-tls\") pod \"maas-api-e2e-worker-w2-9da431-79b57b969c-4n9ts\" (UID: \"f277337a-c920-4c20-a029-215d4ee996f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9da431-79b57b969c-4n9ts" Nov 03 01:31:38.728959 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:31:38.728887 5437 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w2-9da431-serving-cert: secret "maas-api-e2e-worker-w2-9da431-serving-cert" not found Nov 03 01:31:38.728959 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:38.728927 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f277337a-c920-4c20-a029-215d4ee996f9-metrics-certs\") pod \"maas-api-e2e-worker-w2-9da431-79b57b969c-4n9ts\" (UID: \"f277337a-c920-4c20-a029-215d4ee996f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9da431-79b57b969c-4n9ts" Nov 03 01:31:38.728959 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:31:38.728949 5437 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f277337a-c920-4c20-a029-215d4ee996f9-maas-api-tls podName:f277337a-c920-4c20-a029-215d4ee996f9 nodeName:}" failed. No retries permitted until 2026-11-03 01:31:39.228927525 +0000 UTC m=+781.258409812 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f277337a-c920-4c20-a029-215d4ee996f9-maas-api-tls") pod "maas-api-e2e-worker-w2-9da431-79b57b969c-4n9ts" (UID: "f277337a-c920-4c20-a029-215d4ee996f9") : secret "maas-api-e2e-worker-w2-9da431-serving-cert" not found Nov 03 01:31:38.729561 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:38.729536 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f277337a-c920-4c20-a029-215d4ee996f9-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-9da431-79b57b969c-4n9ts\" (UID: \"f277337a-c920-4c20-a029-215d4ee996f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9da431-79b57b969c-4n9ts" Nov 03 01:31:38.731459 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:38.731428 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f277337a-c920-4c20-a029-215d4ee996f9-metrics-certs\") pod \"maas-api-e2e-worker-w2-9da431-79b57b969c-4n9ts\" (UID: \"f277337a-c920-4c20-a029-215d4ee996f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9da431-79b57b969c-4n9ts" Nov 03 01:31:38.796743 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:38.796717 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g682j\" (UniqueName: \"kubernetes.io/projected/f277337a-c920-4c20-a029-215d4ee996f9-kube-api-access-g682j\") pod \"maas-api-e2e-worker-w2-9da431-79b57b969c-4n9ts\" (UID: \"f277337a-c920-4c20-a029-215d4ee996f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9da431-79b57b969c-4n9ts" Nov 03 01:31:38.833886 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:38.833849 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:39.034319 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:39.034277 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:39.234325 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:39.234235 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f277337a-c920-4c20-a029-215d4ee996f9-maas-api-tls\") pod \"maas-api-e2e-worker-w2-9da431-79b57b969c-4n9ts\" (UID: \"f277337a-c920-4c20-a029-215d4ee996f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9da431-79b57b969c-4n9ts" Nov 03 01:31:39.235036 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:39.235000 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:39.237160 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:39.237127 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f277337a-c920-4c20-a029-215d4ee996f9-maas-api-tls\") pod \"maas-api-e2e-worker-w2-9da431-79b57b969c-4n9ts\" (UID: \"f277337a-c920-4c20-a029-215d4ee996f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9da431-79b57b969c-4n9ts" Nov 03 01:31:39.445228 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:39.445197 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9da431-79b57b969c-4n9ts" Nov 03 01:31:39.820297 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:31:39.820264 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf277337a_c920_4c20_a029_215d4ee996f9.slice/crio-8633c0fa33400de290396a94fe48b8fabfd43e5b2af3a22d8d1c35c2a5423720 WatchSource:0}: Error finding container 8633c0fa33400de290396a94fe48b8fabfd43e5b2af3a22d8d1c35c2a5423720: Status 404 returned error can't find the container with id 8633c0fa33400de290396a94fe48b8fabfd43e5b2af3a22d8d1c35c2a5423720 Nov 03 01:31:39.823671 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:39.823639 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-9da431-79b57b969c-4n9ts"] Nov 03 01:31:39.996688 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:39.996654 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:40.064534 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:40.064498 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-41aa5f/simulator-w3-41aa5f-kserve-586cfbcdd4-gsmfk" Nov 03 01:31:40.077467 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:40.077360 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-41aa5f/simulator-w3-41aa5f-kserve-586cfbcdd4-gsmfk" Nov 03 01:31:40.742425 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:40.742386 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e1793d-7f578596cf-fj8br" Nov 03 01:31:40.792093 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:40.792046 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9da431-79b57b969c-4n9ts" event={"ID":"f277337a-c920-4c20-a029-215d4ee996f9","Type":"ContainerStarted","Data":"8633c0fa33400de290396a94fe48b8fabfd43e5b2af3a22d8d1c35c2a5423720"} Nov 03 01:31:40.803977 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:40.803942 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-41aa5f/simulator-w3-41aa5f-kserve-586cfbcdd4-gsmfk" Nov 03 01:31:41.800781 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:41.800738 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9da431-79b57b969c-4n9ts" event={"ID":"f277337a-c920-4c20-a029-215d4ee996f9","Type":"ContainerStarted","Data":"b8132051b75cad7e7ea9ecca9391a203e6876b3c85b236a6930b556b9b35cf62"} Nov 03 01:31:41.801284 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:41.801049 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9da431-79b57b969c-4n9ts" Nov 03 01:31:41.923091 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:41.923027 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9da431-79b57b969c-4n9ts" podStartSLOduration=2.8527821490000003 podStartE2EDuration="3.923011556s" totalImagesPullingTime="1.070229407s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:31:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:31:41.921421358 +0000 UTC m=+783.950903650" watchObservedRunningTime="2026-11-03 01:31:41.923011556 +0000 UTC m=+783.952493855" Nov 03 01:31:43.735999 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:43.735953 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:43.948009 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:43.947974 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:44.035280 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:31:44.035195 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 01:31:44.149553 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:44.149515 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:47.817252 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:47.817220 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9da431-79b57b969c-4n9ts" Nov 03 01:31:48.699591 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:48.699559 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:53.487031 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:53.486990 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:55.190469 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:55.190430 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-0bf34a/simulator-w4-0bf34a-kserve-6bc96559d9-gkdwl"] Nov 03 01:31:55.194313 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:55.194288 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-0bf34a/simulator-w4-0bf34a-kserve-6bc96559d9-gkdwl" Nov 03 01:31:55.249160 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:55.249117 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-0bf34a/simulator-w4-0bf34a-kserve-6bc96559d9-gkdwl"] Nov 03 01:31:55.298977 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:55.298943 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-0bf34a\"/\"simulator-w4-0bf34a-kserve-self-signed-certs\"" Nov 03 01:31:55.298977 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:55.298950 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-0bf34a\"/\"openshift-service-ca.crt\"" Nov 03 01:31:55.300152 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:55.300128 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-0bf34a\"/\"kube-root-ca.crt\"" Nov 03 01:31:55.300276 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:55.300206 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-0bf34a\"/\"default-dockercfg-tk8nr\"" Nov 03 01:31:55.309795 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:55.309756 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prtvj\" (UniqueName: \"kubernetes.io/projected/4cfca4a1-0da6-4637-bc48-8e65776b9ca9-kube-api-access-prtvj\") pod \"simulator-w4-0bf34a-kserve-6bc96559d9-gkdwl\" (UID: \"4cfca4a1-0da6-4637-bc48-8e65776b9ca9\") " pod="e2e-models-e2e-worker-w4-0bf34a/simulator-w4-0bf34a-kserve-6bc96559d9-gkdwl" Nov 03 01:31:55.309922 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:55.309828 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4cfca4a1-0da6-4637-bc48-8e65776b9ca9-tls-certs\") pod \"simulator-w4-0bf34a-kserve-6bc96559d9-gkdwl\" (UID: \"4cfca4a1-0da6-4637-bc48-8e65776b9ca9\") " pod="e2e-models-e2e-worker-w4-0bf34a/simulator-w4-0bf34a-kserve-6bc96559d9-gkdwl" Nov 03 01:31:55.309961 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:55.309920 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cfca4a1-0da6-4637-bc48-8e65776b9ca9-tmp-dir\") pod \"simulator-w4-0bf34a-kserve-6bc96559d9-gkdwl\" (UID: \"4cfca4a1-0da6-4637-bc48-8e65776b9ca9\") " pod="e2e-models-e2e-worker-w4-0bf34a/simulator-w4-0bf34a-kserve-6bc96559d9-gkdwl" Nov 03 01:31:55.309995 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:55.309964 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4cfca4a1-0da6-4637-bc48-8e65776b9ca9-home\") pod \"simulator-w4-0bf34a-kserve-6bc96559d9-gkdwl\" (UID: \"4cfca4a1-0da6-4637-bc48-8e65776b9ca9\") " pod="e2e-models-e2e-worker-w4-0bf34a/simulator-w4-0bf34a-kserve-6bc96559d9-gkdwl" Nov 03 01:31:55.310090 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:55.310065 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4cfca4a1-0da6-4637-bc48-8e65776b9ca9-dshm\") pod \"simulator-w4-0bf34a-kserve-6bc96559d9-gkdwl\" (UID: \"4cfca4a1-0da6-4637-bc48-8e65776b9ca9\") " pod="e2e-models-e2e-worker-w4-0bf34a/simulator-w4-0bf34a-kserve-6bc96559d9-gkdwl" Nov 03 01:31:55.310125 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:55.310099 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4cfca4a1-0da6-4637-bc48-8e65776b9ca9-model-cache\") pod \"simulator-w4-0bf34a-kserve-6bc96559d9-gkdwl\" (UID: \"4cfca4a1-0da6-4637-bc48-8e65776b9ca9\") " pod="e2e-models-e2e-worker-w4-0bf34a/simulator-w4-0bf34a-kserve-6bc96559d9-gkdwl" Nov 03 01:31:55.336531 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:55.336483 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:55.411641 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:55.411596 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4cfca4a1-0da6-4637-bc48-8e65776b9ca9-home\") pod \"simulator-w4-0bf34a-kserve-6bc96559d9-gkdwl\" (UID: \"4cfca4a1-0da6-4637-bc48-8e65776b9ca9\") " pod="e2e-models-e2e-worker-w4-0bf34a/simulator-w4-0bf34a-kserve-6bc96559d9-gkdwl" Nov 03 01:31:55.411854 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:55.411656 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4cfca4a1-0da6-4637-bc48-8e65776b9ca9-dshm\") pod \"simulator-w4-0bf34a-kserve-6bc96559d9-gkdwl\" (UID: \"4cfca4a1-0da6-4637-bc48-8e65776b9ca9\") " pod="e2e-models-e2e-worker-w4-0bf34a/simulator-w4-0bf34a-kserve-6bc96559d9-gkdwl" Nov 03 01:31:55.411854 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:55.411675 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4cfca4a1-0da6-4637-bc48-8e65776b9ca9-model-cache\") pod \"simulator-w4-0bf34a-kserve-6bc96559d9-gkdwl\" (UID: \"4cfca4a1-0da6-4637-bc48-8e65776b9ca9\") " pod="e2e-models-e2e-worker-w4-0bf34a/simulator-w4-0bf34a-kserve-6bc96559d9-gkdwl" Nov 03 01:31:55.411854 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:55.411713 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prtvj\" (UniqueName: \"kubernetes.io/projected/4cfca4a1-0da6-4637-bc48-8e65776b9ca9-kube-api-access-prtvj\") pod \"simulator-w4-0bf34a-kserve-6bc96559d9-gkdwl\" (UID: \"4cfca4a1-0da6-4637-bc48-8e65776b9ca9\") " pod="e2e-models-e2e-worker-w4-0bf34a/simulator-w4-0bf34a-kserve-6bc96559d9-gkdwl" Nov 03 01:31:55.412015 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:55.411867 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4cfca4a1-0da6-4637-bc48-8e65776b9ca9-tls-certs\") pod \"simulator-w4-0bf34a-kserve-6bc96559d9-gkdwl\" (UID: \"4cfca4a1-0da6-4637-bc48-8e65776b9ca9\") " pod="e2e-models-e2e-worker-w4-0bf34a/simulator-w4-0bf34a-kserve-6bc96559d9-gkdwl" Nov 03 01:31:55.412015 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:55.411952 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cfca4a1-0da6-4637-bc48-8e65776b9ca9-tmp-dir\") pod \"simulator-w4-0bf34a-kserve-6bc96559d9-gkdwl\" (UID: \"4cfca4a1-0da6-4637-bc48-8e65776b9ca9\") " pod="e2e-models-e2e-worker-w4-0bf34a/simulator-w4-0bf34a-kserve-6bc96559d9-gkdwl" Nov 03 01:31:55.412120 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:55.412071 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4cfca4a1-0da6-4637-bc48-8e65776b9ca9-home\") pod \"simulator-w4-0bf34a-kserve-6bc96559d9-gkdwl\" (UID: \"4cfca4a1-0da6-4637-bc48-8e65776b9ca9\") " pod="e2e-models-e2e-worker-w4-0bf34a/simulator-w4-0bf34a-kserve-6bc96559d9-gkdwl" Nov 03 01:31:55.412173 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:55.412140 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4cfca4a1-0da6-4637-bc48-8e65776b9ca9-model-cache\") pod \"simulator-w4-0bf34a-kserve-6bc96559d9-gkdwl\" (UID: \"4cfca4a1-0da6-4637-bc48-8e65776b9ca9\") " pod="e2e-models-e2e-worker-w4-0bf34a/simulator-w4-0bf34a-kserve-6bc96559d9-gkdwl" Nov 03 01:31:55.412284 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:55.412264 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cfca4a1-0da6-4637-bc48-8e65776b9ca9-tmp-dir\") pod \"simulator-w4-0bf34a-kserve-6bc96559d9-gkdwl\" (UID: \"4cfca4a1-0da6-4637-bc48-8e65776b9ca9\") " pod="e2e-models-e2e-worker-w4-0bf34a/simulator-w4-0bf34a-kserve-6bc96559d9-gkdwl" Nov 03 01:31:55.414046 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:55.414017 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4cfca4a1-0da6-4637-bc48-8e65776b9ca9-dshm\") pod \"simulator-w4-0bf34a-kserve-6bc96559d9-gkdwl\" (UID: \"4cfca4a1-0da6-4637-bc48-8e65776b9ca9\") " pod="e2e-models-e2e-worker-w4-0bf34a/simulator-w4-0bf34a-kserve-6bc96559d9-gkdwl" Nov 03 01:31:55.414413 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:55.414393 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4cfca4a1-0da6-4637-bc48-8e65776b9ca9-tls-certs\") pod \"simulator-w4-0bf34a-kserve-6bc96559d9-gkdwl\" (UID: \"4cfca4a1-0da6-4637-bc48-8e65776b9ca9\") " pod="e2e-models-e2e-worker-w4-0bf34a/simulator-w4-0bf34a-kserve-6bc96559d9-gkdwl" Nov 03 01:31:55.476997 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:55.476914 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prtvj\" (UniqueName: \"kubernetes.io/projected/4cfca4a1-0da6-4637-bc48-8e65776b9ca9-kube-api-access-prtvj\") pod \"simulator-w4-0bf34a-kserve-6bc96559d9-gkdwl\" (UID: \"4cfca4a1-0da6-4637-bc48-8e65776b9ca9\") " pod="e2e-models-e2e-worker-w4-0bf34a/simulator-w4-0bf34a-kserve-6bc96559d9-gkdwl" Nov 03 01:31:55.506232 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:55.506191 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-0bf34a/simulator-w4-0bf34a-kserve-6bc96559d9-gkdwl" Nov 03 01:31:55.878170 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:31:55.878130 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4cfca4a1_0da6_4637_bc48_8e65776b9ca9.slice/crio-4da70f4d20613b0de5e024066ec6c7f16740d10e46eaaae0dfc5a479e777ef5d WatchSource:0}: Error finding container 4da70f4d20613b0de5e024066ec6c7f16740d10e46eaaae0dfc5a479e777ef5d: Status 404 returned error can't find the container with id 4da70f4d20613b0de5e024066ec6c7f16740d10e46eaaae0dfc5a479e777ef5d Nov 03 01:31:55.880609 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:55.880581 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-0bf34a/simulator-w4-0bf34a-kserve-6bc96559d9-gkdwl"] Nov 03 01:31:55.888908 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:55.888866 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-0bf34a/simulator-w4-0bf34a-kserve-6bc96559d9-gkdwl" event={"ID":"4cfca4a1-0da6-4637-bc48-8e65776b9ca9","Type":"ContainerStarted","Data":"4da70f4d20613b0de5e024066ec6c7f16740d10e46eaaae0dfc5a479e777ef5d"} Nov 03 01:31:56.896340 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:56.896302 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-0bf34a/simulator-w4-0bf34a-kserve-6bc96559d9-gkdwl" event={"ID":"4cfca4a1-0da6-4637-bc48-8e65776b9ca9","Type":"ContainerStarted","Data":"e538f12b29b5f83223c5f12bec4ba1f77361526abf061af9bdbf15490c810688"} Nov 03 01:31:57.016047 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:57.015989 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-0bf34a/simulator-w4-0bf34a-kserve-6bc96559d9-gkdwl" podStartSLOduration=2.015975208 podStartE2EDuration="2.015975208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:31:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:31:57.015906519 +0000 UTC m=+799.045388826" watchObservedRunningTime="2026-11-03 01:31:57.015975208 +0000 UTC m=+799.045457500" Nov 03 01:31:57.507083 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:57.507034 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0bf34a/simulator-w4-0bf34a-kserve-6bc96559d9-gkdwl" Nov 03 01:31:58.378601 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:31:58.378561 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:32:00.815722 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:32:00.815670 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:32:03.083581 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:32:03.083536 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:32:05.507273 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:32:05.507227 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-0bf34a/simulator-w4-0bf34a-kserve-6bc96559d9-gkdwl" Nov 03 01:32:05.520598 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:32:05.520566 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-0bf34a/simulator-w4-0bf34a-kserve-6bc96559d9-gkdwl" Nov 03 01:32:05.968071 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:32:05.968033 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0bf34a/simulator-w4-0bf34a-kserve-6bc96559d9-gkdwl" Nov 03 01:32:07.880303 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:32:07.880264 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:32:10.833028 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:32:10.832982 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-41aa5f/premium-w3-41aa5f-kserve-5fb94fd5fc-gmtkb"] Nov 03 01:32:10.838132 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:32:10.838103 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-41aa5f/premium-w3-41aa5f-kserve-5fb94fd5fc-gmtkb" Nov 03 01:32:10.856012 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:32:10.855969 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c46dca02-6036-46de-b0f3-df796d231f62-tmp-dir\") pod \"premium-w3-41aa5f-kserve-5fb94fd5fc-gmtkb\" (UID: \"c46dca02-6036-46de-b0f3-df796d231f62\") " pod="e2e-models-e2e-worker-w3-41aa5f/premium-w3-41aa5f-kserve-5fb94fd5fc-gmtkb" Nov 03 01:32:10.856012 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:32:10.856015 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c46dca02-6036-46de-b0f3-df796d231f62-dshm\") pod \"premium-w3-41aa5f-kserve-5fb94fd5fc-gmtkb\" (UID: \"c46dca02-6036-46de-b0f3-df796d231f62\") " pod="e2e-models-e2e-worker-w3-41aa5f/premium-w3-41aa5f-kserve-5fb94fd5fc-gmtkb" Nov 03 01:32:10.856246 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:32:10.856114 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c46dca02-6036-46de-b0f3-df796d231f62-home\") pod \"premium-w3-41aa5f-kserve-5fb94fd5fc-gmtkb\" (UID: \"c46dca02-6036-46de-b0f3-df796d231f62\") " pod="e2e-models-e2e-worker-w3-41aa5f/premium-w3-41aa5f-kserve-5fb94fd5fc-gmtkb" Nov 03 01:32:10.856293 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:32:10.856250 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c46dca02-6036-46de-b0f3-df796d231f62-model-cache\") pod \"premium-w3-41aa5f-kserve-5fb94fd5fc-gmtkb\" (UID: \"c46dca02-6036-46de-b0f3-df796d231f62\") " pod="e2e-models-e2e-worker-w3-41aa5f/premium-w3-41aa5f-kserve-5fb94fd5fc-gmtkb" Nov 03 01:32:10.856373 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:32:10.856340 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c46dca02-6036-46de-b0f3-df796d231f62-tls-certs\") pod \"premium-w3-41aa5f-kserve-5fb94fd5fc-gmtkb\" (UID: \"c46dca02-6036-46de-b0f3-df796d231f62\") " pod="e2e-models-e2e-worker-w3-41aa5f/premium-w3-41aa5f-kserve-5fb94fd5fc-gmtkb" Nov 03 01:32:10.856427 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:32:10.856411 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sl78\" (UniqueName: \"kubernetes.io/projected/c46dca02-6036-46de-b0f3-df796d231f62-kube-api-access-4sl78\") pod \"premium-w3-41aa5f-kserve-5fb94fd5fc-gmtkb\" (UID: \"c46dca02-6036-46de-b0f3-df796d231f62\") " pod="e2e-models-e2e-worker-w3-41aa5f/premium-w3-41aa5f-kserve-5fb94fd5fc-gmtkb" Nov 03 01:32:10.893356 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:32:10.893318 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-41aa5f/premium-w3-41aa5f-kserve-5fb94fd5fc-gmtkb"] Nov 03 01:32:10.941742 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:32:10.941705 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-41aa5f\"/\"premium-w3-41aa5f-kserve-self-signed-certs\"" Nov 03 01:32:10.957481 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:32:10.957442 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c46dca02-6036-46de-b0f3-df796d231f62-model-cache\") pod \"premium-w3-41aa5f-kserve-5fb94fd5fc-gmtkb\" (UID: \"c46dca02-6036-46de-b0f3-df796d231f62\") " pod="e2e-models-e2e-worker-w3-41aa5f/premium-w3-41aa5f-kserve-5fb94fd5fc-gmtkb" Nov 03 01:32:10.957642 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:32:10.957494 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c46dca02-6036-46de-b0f3-df796d231f62-tls-certs\") pod \"premium-w3-41aa5f-kserve-5fb94fd5fc-gmtkb\" (UID: \"c46dca02-6036-46de-b0f3-df796d231f62\") " pod="e2e-models-e2e-worker-w3-41aa5f/premium-w3-41aa5f-kserve-5fb94fd5fc-gmtkb" Nov 03 01:32:10.957642 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:32:10.957536 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sl78\" (UniqueName: \"kubernetes.io/projected/c46dca02-6036-46de-b0f3-df796d231f62-kube-api-access-4sl78\") pod \"premium-w3-41aa5f-kserve-5fb94fd5fc-gmtkb\" (UID: \"c46dca02-6036-46de-b0f3-df796d231f62\") " pod="e2e-models-e2e-worker-w3-41aa5f/premium-w3-41aa5f-kserve-5fb94fd5fc-gmtkb" Nov 03 01:32:10.957776 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:32:10.957640 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c46dca02-6036-46de-b0f3-df796d231f62-tmp-dir\") pod \"premium-w3-41aa5f-kserve-5fb94fd5fc-gmtkb\" (UID: \"c46dca02-6036-46de-b0f3-df796d231f62\") " pod="e2e-models-e2e-worker-w3-41aa5f/premium-w3-41aa5f-kserve-5fb94fd5fc-gmtkb" Nov 03 01:32:10.957776 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:32:10.957676 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c46dca02-6036-46de-b0f3-df796d231f62-dshm\") pod \"premium-w3-41aa5f-kserve-5fb94fd5fc-gmtkb\" (UID: \"c46dca02-6036-46de-b0f3-df796d231f62\") " pod="e2e-models-e2e-worker-w3-41aa5f/premium-w3-41aa5f-kserve-5fb94fd5fc-gmtkb" Nov 03 01:32:10.957776 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:32:10.957747 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c46dca02-6036-46de-b0f3-df796d231f62-home\") pod \"premium-w3-41aa5f-kserve-5fb94fd5fc-gmtkb\" (UID: \"c46dca02-6036-46de-b0f3-df796d231f62\") " pod="e2e-models-e2e-worker-w3-41aa5f/premium-w3-41aa5f-kserve-5fb94fd5fc-gmtkb" Nov 03 01:32:10.957956 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:32:10.957932 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c46dca02-6036-46de-b0f3-df796d231f62-model-cache\") pod \"premium-w3-41aa5f-kserve-5fb94fd5fc-gmtkb\" (UID: \"c46dca02-6036-46de-b0f3-df796d231f62\") " pod="e2e-models-e2e-worker-w3-41aa5f/premium-w3-41aa5f-kserve-5fb94fd5fc-gmtkb" Nov 03 01:32:10.958076 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:32:10.958052 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c46dca02-6036-46de-b0f3-df796d231f62-tmp-dir\") pod \"premium-w3-41aa5f-kserve-5fb94fd5fc-gmtkb\" (UID: \"c46dca02-6036-46de-b0f3-df796d231f62\") " pod="e2e-models-e2e-worker-w3-41aa5f/premium-w3-41aa5f-kserve-5fb94fd5fc-gmtkb" Nov 03 01:32:10.958180 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:32:10.958162 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c46dca02-6036-46de-b0f3-df796d231f62-home\") pod \"premium-w3-41aa5f-kserve-5fb94fd5fc-gmtkb\" (UID: \"c46dca02-6036-46de-b0f3-df796d231f62\") " pod="e2e-models-e2e-worker-w3-41aa5f/premium-w3-41aa5f-kserve-5fb94fd5fc-gmtkb" Nov 03 01:32:10.959976 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:32:10.959932 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c46dca02-6036-46de-b0f3-df796d231f62-dshm\") pod \"premium-w3-41aa5f-kserve-5fb94fd5fc-gmtkb\" (UID: \"c46dca02-6036-46de-b0f3-df796d231f62\") " pod="e2e-models-e2e-worker-w3-41aa5f/premium-w3-41aa5f-kserve-5fb94fd5fc-gmtkb" Nov 03 01:32:10.960149 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:32:10.960129 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c46dca02-6036-46de-b0f3-df796d231f62-tls-certs\") pod \"premium-w3-41aa5f-kserve-5fb94fd5fc-gmtkb\" (UID: \"c46dca02-6036-46de-b0f3-df796d231f62\") " pod="e2e-models-e2e-worker-w3-41aa5f/premium-w3-41aa5f-kserve-5fb94fd5fc-gmtkb" Nov 03 01:32:10.981808 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:32:10.981775 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:32:11.020754 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:32:11.020716 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sl78\" (UniqueName: \"kubernetes.io/projected/c46dca02-6036-46de-b0f3-df796d231f62-kube-api-access-4sl78\") pod \"premium-w3-41aa5f-kserve-5fb94fd5fc-gmtkb\" (UID: \"c46dca02-6036-46de-b0f3-df796d231f62\") " pod="e2e-models-e2e-worker-w3-41aa5f/premium-w3-41aa5f-kserve-5fb94fd5fc-gmtkb" Nov 03 01:32:11.183167 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:32:11.183081 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-41aa5f/premium-w3-41aa5f-kserve-5fb94fd5fc-gmtkb" Nov 03 01:32:11.555492 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:32:11.555458 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc46dca02_6036_46de_b0f3_df796d231f62.slice/crio-c59de78b693f8e4a644b8940c602fb78dffbcef6d6d9e4121cc6a1b346f3054e WatchSource:0}: Error finding container c59de78b693f8e4a644b8940c602fb78dffbcef6d6d9e4121cc6a1b346f3054e: Status 404 returned error can't find the container with id c59de78b693f8e4a644b8940c602fb78dffbcef6d6d9e4121cc6a1b346f3054e Nov 03 01:32:11.558277 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:32:11.558197 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-41aa5f/premium-w3-41aa5f-kserve-5fb94fd5fc-gmtkb"] Nov 03 01:32:11.985930 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:32:11.985822 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-41aa5f/premium-w3-41aa5f-kserve-5fb94fd5fc-gmtkb" event={"ID":"c46dca02-6036-46de-b0f3-df796d231f62","Type":"ContainerStarted","Data":"a4bacbf873e67a41775fb4df36c84815925b00fb28290cd91cbcc237f594249f"} Nov 03 01:32:11.985930 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:32:11.985874 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-41aa5f/premium-w3-41aa5f-kserve-5fb94fd5fc-gmtkb" event={"ID":"c46dca02-6036-46de-b0f3-df796d231f62","Type":"ContainerStarted","Data":"c59de78b693f8e4a644b8940c602fb78dffbcef6d6d9e4121cc6a1b346f3054e"} Nov 03 01:32:12.108584 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:32:12.108515 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-41aa5f/premium-w3-41aa5f-kserve-5fb94fd5fc-gmtkb" podStartSLOduration=2.108494011 podStartE2EDuration="2.108494011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:32:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:32:12.106222961 +0000 UTC m=+814.135705256" watchObservedRunningTime="2026-11-03 01:32:12.108494011 +0000 UTC m=+814.137976307" Nov 03 01:32:12.183838 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:32:12.183799 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-41aa5f/premium-w3-41aa5f-kserve-5fb94fd5fc-gmtkb" Nov 03 01:32:12.750396 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:32:12.750341 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:32:13.619353 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:32:13.619317 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:32:14.036177 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:32:14.036095 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 01:32:14.223647 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:32:14.223612 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:32:14.432321 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:32:14.432283 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:32:17.779990 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:32:17.779903 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:32:19.746596 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:32:19.746557 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:32:20.257844 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:32:20.257805 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:32:20.459287 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:32:20.459227 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:32:21.183635 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:32:21.183603 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-41aa5f/premium-w3-41aa5f-kserve-5fb94fd5fc-gmtkb" Nov 03 01:32:21.196131 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:32:21.196105 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-41aa5f/premium-w3-41aa5f-kserve-5fb94fd5fc-gmtkb" Nov 03 01:32:22.049100 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:32:22.049069 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-41aa5f/premium-w3-41aa5f-kserve-5fb94fd5fc-gmtkb" Nov 03 01:32:22.689247 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:32:22.689205 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:32:27.464726 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:32:27.464685 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:32:32.183832 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:32:32.183789 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:32:37.083746 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:32:37.083706 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:32:39.791830 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:32:39.791793 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:32:40.339078 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:32:40.339039 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:32:41.890339 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:32:41.890295 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:32:44.034097 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:32:44.034056 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 01:32:44.810235 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:32:44.810193 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:32:46.674603 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:32:46.674568 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:32:50.452189 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:32:50.452152 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-e1793d/simulator-w1-e1793d-kserve-7595dfc6b7-dbbgf"] Nov 03 01:32:50.457205 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:32:50.457187 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e1793d/simulator-w1-e1793d-kserve-7595dfc6b7-dbbgf" Nov 03 01:32:50.512185 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:32:50.512149 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-e1793d/simulator-w1-e1793d-kserve-7595dfc6b7-dbbgf"] Nov 03 01:32:50.561572 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:32:50.561528 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-e1793d\"/\"openshift-service-ca.crt\"" Nov 03 01:32:50.561756 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:32:50.561587 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-e1793d\"/\"default-dockercfg-s7nrz\"" Nov 03 01:32:50.562225 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:32:50.562207 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f28ccf15-b92d-400c-9204-69b66279d7d8-tmp-dir\") pod \"simulator-w1-e1793d-kserve-7595dfc6b7-dbbgf\" (UID: \"f28ccf15-b92d-400c-9204-69b66279d7d8\") " pod="e2e-models-e2e-worker-w1-e1793d/simulator-w1-e1793d-kserve-7595dfc6b7-dbbgf" Nov 03 01:32:50.562270 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:32:50.562237 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f28ccf15-b92d-400c-9204-69b66279d7d8-home\") pod \"simulator-w1-e1793d-kserve-7595dfc6b7-dbbgf\" (UID: \"f28ccf15-b92d-400c-9204-69b66279d7d8\") " pod="e2e-models-e2e-worker-w1-e1793d/simulator-w1-e1793d-kserve-7595dfc6b7-dbbgf" Nov 03 01:32:50.562359 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:32:50.562346 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f28ccf15-b92d-400c-9204-69b66279d7d8-model-cache\") pod \"simulator-w1-e1793d-kserve-7595dfc6b7-dbbgf\" (UID: \"f28ccf15-b92d-400c-9204-69b66279d7d8\") " pod="e2e-models-e2e-worker-w1-e1793d/simulator-w1-e1793d-kserve-7595dfc6b7-dbbgf" Nov 03 01:32:50.562437 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:32:50.562397 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f28ccf15-b92d-400c-9204-69b66279d7d8-dshm\") pod \"simulator-w1-e1793d-kserve-7595dfc6b7-dbbgf\" (UID: \"f28ccf15-b92d-400c-9204-69b66279d7d8\") " pod="e2e-models-e2e-worker-w1-e1793d/simulator-w1-e1793d-kserve-7595dfc6b7-dbbgf" Nov 03 01:32:50.562510 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:32:50.562487 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtfnh\" (UniqueName: \"kubernetes.io/projected/f28ccf15-b92d-400c-9204-69b66279d7d8-kube-api-access-rtfnh\") pod \"simulator-w1-e1793d-kserve-7595dfc6b7-dbbgf\" (UID: \"f28ccf15-b92d-400c-9204-69b66279d7d8\") " pod="e2e-models-e2e-worker-w1-e1793d/simulator-w1-e1793d-kserve-7595dfc6b7-dbbgf" Nov 03 01:32:50.562554 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:32:50.562542 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f28ccf15-b92d-400c-9204-69b66279d7d8-tls-certs\") pod \"simulator-w1-e1793d-kserve-7595dfc6b7-dbbgf\" (UID: \"f28ccf15-b92d-400c-9204-69b66279d7d8\") " pod="e2e-models-e2e-worker-w1-e1793d/simulator-w1-e1793d-kserve-7595dfc6b7-dbbgf" Nov 03 01:32:50.563167 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:32:50.563151 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-e1793d\"/\"kube-root-ca.crt\"" Nov 03 01:32:50.563256 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:32:50.563241 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-e1793d\"/\"simulator-w1-e1793d-kserve-self-signed-certs\"" Nov 03 01:32:50.598651 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:32:50.598616 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:32:50.663472 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:32:50.663441 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f28ccf15-b92d-400c-9204-69b66279d7d8-model-cache\") pod \"simulator-w1-e1793d-kserve-7595dfc6b7-dbbgf\" (UID: \"f28ccf15-b92d-400c-9204-69b66279d7d8\") " pod="e2e-models-e2e-worker-w1-e1793d/simulator-w1-e1793d-kserve-7595dfc6b7-dbbgf" Nov 03 01:32:50.663665 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:32:50.663488 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f28ccf15-b92d-400c-9204-69b66279d7d8-dshm\") pod \"simulator-w1-e1793d-kserve-7595dfc6b7-dbbgf\" (UID: \"f28ccf15-b92d-400c-9204-69b66279d7d8\") " pod="e2e-models-e2e-worker-w1-e1793d/simulator-w1-e1793d-kserve-7595dfc6b7-dbbgf" Nov 03 01:32:50.663665 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:32:50.663526 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtfnh\" (UniqueName: \"kubernetes.io/projected/f28ccf15-b92d-400c-9204-69b66279d7d8-kube-api-access-rtfnh\") pod \"simulator-w1-e1793d-kserve-7595dfc6b7-dbbgf\" (UID: \"f28ccf15-b92d-400c-9204-69b66279d7d8\") " pod="e2e-models-e2e-worker-w1-e1793d/simulator-w1-e1793d-kserve-7595dfc6b7-dbbgf" Nov 03 01:32:50.663665 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:32:50.663556 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f28ccf15-b92d-400c-9204-69b66279d7d8-tls-certs\") pod \"simulator-w1-e1793d-kserve-7595dfc6b7-dbbgf\" (UID: \"f28ccf15-b92d-400c-9204-69b66279d7d8\") " pod="e2e-models-e2e-worker-w1-e1793d/simulator-w1-e1793d-kserve-7595dfc6b7-dbbgf" Nov 03 01:32:50.663665 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:32:50.663602 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f28ccf15-b92d-400c-9204-69b66279d7d8-tmp-dir\") pod \"simulator-w1-e1793d-kserve-7595dfc6b7-dbbgf\" (UID: \"f28ccf15-b92d-400c-9204-69b66279d7d8\") " pod="e2e-models-e2e-worker-w1-e1793d/simulator-w1-e1793d-kserve-7595dfc6b7-dbbgf" Nov 03 01:32:50.663665 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:32:50.663627 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f28ccf15-b92d-400c-9204-69b66279d7d8-home\") pod \"simulator-w1-e1793d-kserve-7595dfc6b7-dbbgf\" (UID: \"f28ccf15-b92d-400c-9204-69b66279d7d8\") " pod="e2e-models-e2e-worker-w1-e1793d/simulator-w1-e1793d-kserve-7595dfc6b7-dbbgf" Nov 03 01:32:50.663939 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:32:50.663839 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f28ccf15-b92d-400c-9204-69b66279d7d8-model-cache\") pod \"simulator-w1-e1793d-kserve-7595dfc6b7-dbbgf\" (UID: \"f28ccf15-b92d-400c-9204-69b66279d7d8\") " pod="e2e-models-e2e-worker-w1-e1793d/simulator-w1-e1793d-kserve-7595dfc6b7-dbbgf" Nov 03 01:32:50.663997 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:32:50.663930 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f28ccf15-b92d-400c-9204-69b66279d7d8-home\") pod \"simulator-w1-e1793d-kserve-7595dfc6b7-dbbgf\" (UID: \"f28ccf15-b92d-400c-9204-69b66279d7d8\") " pod="e2e-models-e2e-worker-w1-e1793d/simulator-w1-e1793d-kserve-7595dfc6b7-dbbgf" Nov 03 01:32:50.663997 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:32:50.663986 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f28ccf15-b92d-400c-9204-69b66279d7d8-tmp-dir\") pod \"simulator-w1-e1793d-kserve-7595dfc6b7-dbbgf\" (UID: \"f28ccf15-b92d-400c-9204-69b66279d7d8\") " pod="e2e-models-e2e-worker-w1-e1793d/simulator-w1-e1793d-kserve-7595dfc6b7-dbbgf" Nov 03 01:32:50.665695 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:32:50.665670 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f28ccf15-b92d-400c-9204-69b66279d7d8-dshm\") pod \"simulator-w1-e1793d-kserve-7595dfc6b7-dbbgf\" (UID: \"f28ccf15-b92d-400c-9204-69b66279d7d8\") " pod="e2e-models-e2e-worker-w1-e1793d/simulator-w1-e1793d-kserve-7595dfc6b7-dbbgf" Nov 03 01:32:50.666049 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:32:50.666031 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f28ccf15-b92d-400c-9204-69b66279d7d8-tls-certs\") pod \"simulator-w1-e1793d-kserve-7595dfc6b7-dbbgf\" (UID: \"f28ccf15-b92d-400c-9204-69b66279d7d8\") " pod="e2e-models-e2e-worker-w1-e1793d/simulator-w1-e1793d-kserve-7595dfc6b7-dbbgf" Nov 03 01:32:50.730726 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:32:50.730653 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtfnh\" (UniqueName: \"kubernetes.io/projected/f28ccf15-b92d-400c-9204-69b66279d7d8-kube-api-access-rtfnh\") pod \"simulator-w1-e1793d-kserve-7595dfc6b7-dbbgf\" (UID: \"f28ccf15-b92d-400c-9204-69b66279d7d8\") " pod="e2e-models-e2e-worker-w1-e1793d/simulator-w1-e1793d-kserve-7595dfc6b7-dbbgf" Nov 03 01:32:50.769590 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:32:50.769556 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e1793d/simulator-w1-e1793d-kserve-7595dfc6b7-dbbgf" Nov 03 01:32:51.142424 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:32:51.142389 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf28ccf15_b92d_400c_9204_69b66279d7d8.slice/crio-8559ea3d503bd3c5ae2926cdedfa394367971561b4ba3a150e69bcc4c7c5ded1 WatchSource:0}: Error finding container 8559ea3d503bd3c5ae2926cdedfa394367971561b4ba3a150e69bcc4c7c5ded1: Status 404 returned error can't find the container with id 8559ea3d503bd3c5ae2926cdedfa394367971561b4ba3a150e69bcc4c7c5ded1 Nov 03 01:32:51.145114 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:32:51.145089 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-e1793d/simulator-w1-e1793d-kserve-7595dfc6b7-dbbgf"] Nov 03 01:32:51.186794 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:32:51.186766 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e1793d/simulator-w1-e1793d-kserve-7595dfc6b7-dbbgf" event={"ID":"f28ccf15-b92d-400c-9204-69b66279d7d8","Type":"ContainerStarted","Data":"8559ea3d503bd3c5ae2926cdedfa394367971561b4ba3a150e69bcc4c7c5ded1"} Nov 03 01:32:51.489624 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:32:51.489532 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:32:52.193860 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:32:52.193824 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e1793d/simulator-w1-e1793d-kserve-7595dfc6b7-dbbgf" event={"ID":"f28ccf15-b92d-400c-9204-69b66279d7d8","Type":"ContainerStarted","Data":"5bb9895276de4d8c7039a44493615e6ccf353375f7c04526e05b468626216b2d"} Nov 03 01:32:52.316137 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:32:52.316086 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-e1793d/simulator-w1-e1793d-kserve-7595dfc6b7-dbbgf" podStartSLOduration=2.316069474 podStartE2EDuration="2.316069474s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:32:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:32:52.314598688 +0000 UTC m=+854.344080985" watchObservedRunningTime="2026-11-03 01:32:52.316069474 +0000 UTC m=+854.345551813" Nov 03 01:32:52.770036 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:32:52.770001 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e1793d/simulator-w1-e1793d-kserve-7595dfc6b7-dbbgf" Nov 03 01:32:56.212256 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:32:56.212220 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:33:00.770525 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:33:00.770492 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-e1793d/simulator-w1-e1793d-kserve-7595dfc6b7-dbbgf" Nov 03 01:33:00.782936 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:33:00.782903 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-e1793d/simulator-w1-e1793d-kserve-7595dfc6b7-dbbgf" Nov 03 01:33:00.977344 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:33:00.977308 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:33:01.252082 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:33:01.252054 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e1793d/simulator-w1-e1793d-kserve-7595dfc6b7-dbbgf" Nov 03 01:33:04.111729 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:33:04.111693 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:33:05.727349 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:33:05.727310 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-wgtp9"] Nov 03 01:33:05.731190 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:33:05.731169 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-wgtp9" Nov 03 01:33:05.787092 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:33:05.787052 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-41aa5f-openshift-default-dockercfg-g26tf\"" Nov 03 01:33:05.789297 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:33:05.789261 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-wgtp9"] Nov 03 01:33:05.797974 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:33:05.797943 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:33:05.828108 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:33:05.828075 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6807c987-7fd9-4b0d-bee4-dab5799ece82-istio-envoy\") pod \"e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-wgtp9\" (UID: \"6807c987-7fd9-4b0d-bee4-dab5799ece82\") " pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-wgtp9" Nov 03 01:33:05.828300 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:33:05.828125 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6807c987-7fd9-4b0d-bee4-dab5799ece82-credential-socket\") pod \"e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-wgtp9\" (UID: \"6807c987-7fd9-4b0d-bee4-dab5799ece82\") " pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-wgtp9" Nov 03 01:33:05.828300 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:33:05.828145 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6807c987-7fd9-4b0d-bee4-dab5799ece82-istiod-ca-cert\") pod \"e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-wgtp9\" (UID: \"6807c987-7fd9-4b0d-bee4-dab5799ece82\") " pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-wgtp9" Nov 03 01:33:05.828300 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:33:05.828288 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6807c987-7fd9-4b0d-bee4-dab5799ece82-istio-data\") pod \"e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-wgtp9\" (UID: \"6807c987-7fd9-4b0d-bee4-dab5799ece82\") " pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-wgtp9" Nov 03 01:33:05.828470 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:33:05.828328 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6807c987-7fd9-4b0d-bee4-dab5799ece82-istio-podinfo\") pod \"e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-wgtp9\" (UID: \"6807c987-7fd9-4b0d-bee4-dab5799ece82\") " pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-wgtp9" Nov 03 01:33:05.828470 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:33:05.828345 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6807c987-7fd9-4b0d-bee4-dab5799ece82-istio-token\") pod \"e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-wgtp9\" (UID: \"6807c987-7fd9-4b0d-bee4-dab5799ece82\") " pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-wgtp9" Nov 03 01:33:05.828470 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:33:05.828388 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6807c987-7fd9-4b0d-bee4-dab5799ece82-workload-certs\") pod \"e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-wgtp9\" (UID: \"6807c987-7fd9-4b0d-bee4-dab5799ece82\") " pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-wgtp9" Nov 03 01:33:05.828575 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:33:05.828460 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6807c987-7fd9-4b0d-bee4-dab5799ece82-workload-socket\") pod \"e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-wgtp9\" (UID: \"6807c987-7fd9-4b0d-bee4-dab5799ece82\") " pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-wgtp9" Nov 03 01:33:05.828575 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:33:05.828532 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ks474\" (UniqueName: \"kubernetes.io/projected/6807c987-7fd9-4b0d-bee4-dab5799ece82-kube-api-access-ks474\") pod \"e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-wgtp9\" (UID: \"6807c987-7fd9-4b0d-bee4-dab5799ece82\") " pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-wgtp9" Nov 03 01:33:05.931103 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:33:05.931063 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ks474\" (UniqueName: \"kubernetes.io/projected/6807c987-7fd9-4b0d-bee4-dab5799ece82-kube-api-access-ks474\") pod \"e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-wgtp9\" (UID: \"6807c987-7fd9-4b0d-bee4-dab5799ece82\") " pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-wgtp9" Nov 03 01:33:05.931302 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:33:05.931198 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6807c987-7fd9-4b0d-bee4-dab5799ece82-istio-envoy\") pod \"e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-wgtp9\" (UID: \"6807c987-7fd9-4b0d-bee4-dab5799ece82\") " pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-wgtp9" Nov 03 01:33:05.931302 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:33:05.931241 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6807c987-7fd9-4b0d-bee4-dab5799ece82-credential-socket\") pod \"e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-wgtp9\" (UID: \"6807c987-7fd9-4b0d-bee4-dab5799ece82\") " pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-wgtp9" Nov 03 01:33:05.931302 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:33:05.931264 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6807c987-7fd9-4b0d-bee4-dab5799ece82-istiod-ca-cert\") pod \"e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-wgtp9\" (UID: \"6807c987-7fd9-4b0d-bee4-dab5799ece82\") " pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-wgtp9" Nov 03 01:33:05.931302 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:33:05.931301 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6807c987-7fd9-4b0d-bee4-dab5799ece82-istio-data\") pod \"e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-wgtp9\" (UID: \"6807c987-7fd9-4b0d-bee4-dab5799ece82\") " pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-wgtp9" Nov 03 01:33:05.931577 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:33:05.931334 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6807c987-7fd9-4b0d-bee4-dab5799ece82-istio-podinfo\") pod \"e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-wgtp9\" (UID: \"6807c987-7fd9-4b0d-bee4-dab5799ece82\") " pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-wgtp9" Nov 03 01:33:05.931577 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:33:05.931358 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6807c987-7fd9-4b0d-bee4-dab5799ece82-istio-token\") pod \"e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-wgtp9\" (UID: \"6807c987-7fd9-4b0d-bee4-dab5799ece82\") " pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-wgtp9" Nov 03 01:33:05.931577 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:33:05.931430 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6807c987-7fd9-4b0d-bee4-dab5799ece82-workload-certs\") pod \"e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-wgtp9\" (UID: \"6807c987-7fd9-4b0d-bee4-dab5799ece82\") " pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-wgtp9" Nov 03 01:33:05.931577 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:33:05.931456 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6807c987-7fd9-4b0d-bee4-dab5799ece82-workload-socket\") pod \"e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-wgtp9\" (UID: \"6807c987-7fd9-4b0d-bee4-dab5799ece82\") " pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-wgtp9" Nov 03 01:33:05.931834 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:33:05.931805 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6807c987-7fd9-4b0d-bee4-dab5799ece82-istio-data\") pod \"e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-wgtp9\" (UID: \"6807c987-7fd9-4b0d-bee4-dab5799ece82\") " pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-wgtp9" Nov 03 01:33:05.932010 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:33:05.931846 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6807c987-7fd9-4b0d-bee4-dab5799ece82-workload-socket\") pod \"e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-wgtp9\" (UID: \"6807c987-7fd9-4b0d-bee4-dab5799ece82\") " pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-wgtp9" Nov 03 01:33:05.932136 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:33:05.932109 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6807c987-7fd9-4b0d-bee4-dab5799ece82-credential-socket\") pod \"e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-wgtp9\" (UID: \"6807c987-7fd9-4b0d-bee4-dab5799ece82\") " pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-wgtp9" Nov 03 01:33:05.932136 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:33:05.932125 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6807c987-7fd9-4b0d-bee4-dab5799ece82-workload-certs\") pod \"e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-wgtp9\" (UID: \"6807c987-7fd9-4b0d-bee4-dab5799ece82\") " pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-wgtp9" Nov 03 01:33:05.932652 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:33:05.932628 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6807c987-7fd9-4b0d-bee4-dab5799ece82-istiod-ca-cert\") pod \"e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-wgtp9\" (UID: \"6807c987-7fd9-4b0d-bee4-dab5799ece82\") " pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-wgtp9" Nov 03 01:33:05.934333 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:33:05.934299 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6807c987-7fd9-4b0d-bee4-dab5799ece82-istio-envoy\") pod \"e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-wgtp9\" (UID: \"6807c987-7fd9-4b0d-bee4-dab5799ece82\") " pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-wgtp9" Nov 03 01:33:05.934448 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:33:05.934387 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6807c987-7fd9-4b0d-bee4-dab5799ece82-istio-podinfo\") pod \"e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-wgtp9\" (UID: \"6807c987-7fd9-4b0d-bee4-dab5799ece82\") " pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-wgtp9" Nov 03 01:33:05.994480 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:33:05.994396 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6807c987-7fd9-4b0d-bee4-dab5799ece82-istio-token\") pod \"e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-wgtp9\" (UID: \"6807c987-7fd9-4b0d-bee4-dab5799ece82\") " pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-wgtp9" Nov 03 01:33:05.994703 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:33:05.994676 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ks474\" (UniqueName: \"kubernetes.io/projected/6807c987-7fd9-4b0d-bee4-dab5799ece82-kube-api-access-ks474\") pod \"e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-wgtp9\" (UID: \"6807c987-7fd9-4b0d-bee4-dab5799ece82\") " pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-wgtp9" Nov 03 01:33:06.044756 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:33:06.044721 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-wgtp9" Nov 03 01:33:06.257476 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:33:06.257374 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:33:06.476645 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:33:06.476608 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6807c987_7fd9_4b0d_bee4_dab5799ece82.slice/crio-f6664f318d0507e42da429d7e25bf7d6ffabf4e0a68812403b4d627642f62ac2 WatchSource:0}: Error finding container f6664f318d0507e42da429d7e25bf7d6ffabf4e0a68812403b4d627642f62ac2: Status 404 returned error can't find the container with id f6664f318d0507e42da429d7e25bf7d6ffabf4e0a68812403b4d627642f62ac2 Nov 03 01:33:06.479199 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:33:06.479164 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 01:33:06.479285 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:33:06.479233 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 01:33:06.479285 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:33:06.479264 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 01:33:06.479355 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:33:06.479328 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-wgtp9"] Nov 03 01:33:07.276348 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:33:07.276311 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-wgtp9" event={"ID":"6807c987-7fd9-4b0d-bee4-dab5799ece82","Type":"ContainerStarted","Data":"421af643089b5b8096e4e1f8fbac7c9e1f8409c5ef2bf3134677aaec3660fc8d"} Nov 03 01:33:07.276838 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:33:07.276393 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-wgtp9" event={"ID":"6807c987-7fd9-4b0d-bee4-dab5799ece82","Type":"ContainerStarted","Data":"f6664f318d0507e42da429d7e25bf7d6ffabf4e0a68812403b4d627642f62ac2"} Nov 03 01:33:07.452438 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:33:07.452352 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-wgtp9" podStartSLOduration=2.4523374909999998 podStartE2EDuration="2.452337491s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:33:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:33:07.451172128 +0000 UTC m=+869.480654423" watchObservedRunningTime="2026-11-03 01:33:07.452337491 +0000 UTC m=+869.481819786" Nov 03 01:33:08.045780 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:33:08.045733 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-wgtp9" Nov 03 01:33:08.047730 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:33:08.047700 5437 patch_prober.go:29] interesting pod/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-wgtp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Nov 03 01:33:08.047864 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:33:08.047750 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-wgtp9" podUID="6807c987-7fd9-4b0d-bee4-dab5799ece82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Nov 03 01:33:09.045706 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:33:09.045664 5437 patch_prober.go:29] interesting pod/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-wgtp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Nov 03 01:33:09.046173 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:33:09.045727 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-wgtp9" podUID="6807c987-7fd9-4b0d-bee4-dab5799ece82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Nov 03 01:33:10.045934 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:33:10.045894 5437 patch_prober.go:29] interesting pod/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-wgtp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Nov 03 01:33:10.046357 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:33:10.045961 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-wgtp9" podUID="6807c987-7fd9-4b0d-bee4-dab5799ece82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Nov 03 01:33:10.642388 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:33:10.639006 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:33:11.055464 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:33:11.046573 5437 patch_prober.go:29] interesting pod/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-wgtp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Nov 03 01:33:11.055464 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:33:11.046636 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-wgtp9" podUID="6807c987-7fd9-4b0d-bee4-dab5799ece82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Nov 03 01:33:12.045858 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:33:12.045810 5437 patch_prober.go:29] interesting pod/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-wgtp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Nov 03 01:33:12.046046 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:33:12.045883 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-wgtp9" podUID="6807c987-7fd9-4b0d-bee4-dab5799ece82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Nov 03 01:33:13.794815 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:33:13.794785 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-wgtp9" Nov 03 01:33:13.795421 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:33:13.795388 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-wgtp9" Nov 03 01:33:13.796357 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:33:13.796330 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-wgtp9" Nov 03 01:33:14.032917 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:33:14.032876 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 01:33:15.383076 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:33:15.383029 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:33:20.185030 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:33:20.184991 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:33:24.984350 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:33:24.984312 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:33:29.794258 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:33:29.794199 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:33:34.584432 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:33:34.584393 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:33:37.497510 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:33:37.497467 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:33:39.388500 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:33:39.388460 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:33:40.669292 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:33:40.669258 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:33:44.044061 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:33:44.044012 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 01:33:44.174873 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:33:44.174834 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:33:49.000505 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:33:49.000419 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:33:51.373985 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:33:51.373946 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:33:53.792178 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:33:53.792132 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:33:58.573150 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:33:58.573106 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:34:03.391847 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:34:03.391802 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:34:04.920817 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:34:04.920765 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:34:08.194570 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:34:08.194523 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:34:13.143107 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:34:13.143066 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:34:14.036073 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:34:14.036034 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 01:34:17.873936 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:34:17.873887 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:34:22.663717 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:34:22.663658 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:34:27.523133 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:34:27.523092 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:34:28.422128 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:34:28.422088 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:34:28.632210 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:34:28.632171 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:34:32.385475 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:34:32.385431 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:34:32.640566 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:34:32.640471 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:34:37.180222 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:34:37.180182 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:34:41.018038 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:34:41.018006 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:34:41.991352 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:34:41.991309 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:34:44.036076 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:34:44.036036 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 01:34:46.724024 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:34:46.723965 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:34:51.574541 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:34:51.574493 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:34:55.510539 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:34:55.510495 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:34:55.706940 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:34:55.706904 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:34:56.296720 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:34:56.296678 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:34:56.549771 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:34:56.549670 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:35:01.292078 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:01.292010 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:35:04.147712 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:04.147676 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-2w7h6"] Nov 03 01:35:04.151618 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:04.151593 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-2w7h6" Nov 03 01:35:04.230500 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:04.230451 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aa3545f-2866-4899-8e50-1b55db8a4b1b-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-2w7h6\" (UID: \"1aa3545f-2866-4899-8e50-1b55db8a4b1b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-2w7h6" Nov 03 01:35:04.230696 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:04.230512 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1aa3545f-2866-4899-8e50-1b55db8a4b1b-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-2w7h6\" (UID: \"1aa3545f-2866-4899-8e50-1b55db8a4b1b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-2w7h6" Nov 03 01:35:04.230696 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:04.230633 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1aa3545f-2866-4899-8e50-1b55db8a4b1b-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-2w7h6\" (UID: \"1aa3545f-2866-4899-8e50-1b55db8a4b1b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-2w7h6" Nov 03 01:35:04.230696 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:04.230687 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46n8n\" (UniqueName: \"kubernetes.io/projected/1aa3545f-2866-4899-8e50-1b55db8a4b1b-kube-api-access-46n8n\") pod \"maas-default-gateway-openshift-default-59f957fdd7-2w7h6\" (UID: \"1aa3545f-2866-4899-8e50-1b55db8a4b1b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-2w7h6" Nov 03 01:35:04.230824 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:04.230707 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1aa3545f-2866-4899-8e50-1b55db8a4b1b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-2w7h6\" (UID: \"1aa3545f-2866-4899-8e50-1b55db8a4b1b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-2w7h6" Nov 03 01:35:04.230824 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:04.230735 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1aa3545f-2866-4899-8e50-1b55db8a4b1b-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-2w7h6\" (UID: \"1aa3545f-2866-4899-8e50-1b55db8a4b1b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-2w7h6" Nov 03 01:35:04.230888 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:04.230823 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aa3545f-2866-4899-8e50-1b55db8a4b1b-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-2w7h6\" (UID: \"1aa3545f-2866-4899-8e50-1b55db8a4b1b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-2w7h6" Nov 03 01:35:04.230928 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:04.230912 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1aa3545f-2866-4899-8e50-1b55db8a4b1b-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-2w7h6\" (UID: \"1aa3545f-2866-4899-8e50-1b55db8a4b1b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-2w7h6" Nov 03 01:35:04.230964 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:04.230947 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1aa3545f-2866-4899-8e50-1b55db8a4b1b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-2w7h6\" (UID: \"1aa3545f-2866-4899-8e50-1b55db8a4b1b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-2w7h6" Nov 03 01:35:04.276504 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:04.276464 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-2w7h6"] Nov 03 01:35:04.332243 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:04.332201 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1aa3545f-2866-4899-8e50-1b55db8a4b1b-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-2w7h6\" (UID: \"1aa3545f-2866-4899-8e50-1b55db8a4b1b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-2w7h6" Nov 03 01:35:04.332243 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:04.332246 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1aa3545f-2866-4899-8e50-1b55db8a4b1b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-2w7h6\" (UID: \"1aa3545f-2866-4899-8e50-1b55db8a4b1b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-2w7h6" Nov 03 01:35:04.332550 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:04.332286 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aa3545f-2866-4899-8e50-1b55db8a4b1b-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-2w7h6\" (UID: \"1aa3545f-2866-4899-8e50-1b55db8a4b1b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-2w7h6" Nov 03 01:35:04.332550 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:04.332328 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1aa3545f-2866-4899-8e50-1b55db8a4b1b-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-2w7h6\" (UID: \"1aa3545f-2866-4899-8e50-1b55db8a4b1b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-2w7h6" Nov 03 01:35:04.332550 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:04.332500 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1aa3545f-2866-4899-8e50-1b55db8a4b1b-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-2w7h6\" (UID: \"1aa3545f-2866-4899-8e50-1b55db8a4b1b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-2w7h6" Nov 03 01:35:04.332720 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:04.332567 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46n8n\" (UniqueName: \"kubernetes.io/projected/1aa3545f-2866-4899-8e50-1b55db8a4b1b-kube-api-access-46n8n\") pod \"maas-default-gateway-openshift-default-59f957fdd7-2w7h6\" (UID: \"1aa3545f-2866-4899-8e50-1b55db8a4b1b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-2w7h6" Nov 03 01:35:04.332720 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:04.332595 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1aa3545f-2866-4899-8e50-1b55db8a4b1b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-2w7h6\" (UID: \"1aa3545f-2866-4899-8e50-1b55db8a4b1b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-2w7h6" Nov 03 01:35:04.332720 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:04.332639 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1aa3545f-2866-4899-8e50-1b55db8a4b1b-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-2w7h6\" (UID: \"1aa3545f-2866-4899-8e50-1b55db8a4b1b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-2w7h6" Nov 03 01:35:04.332720 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:04.332693 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aa3545f-2866-4899-8e50-1b55db8a4b1b-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-2w7h6\" (UID: \"1aa3545f-2866-4899-8e50-1b55db8a4b1b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-2w7h6" Nov 03 01:35:04.332920 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:04.332713 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1aa3545f-2866-4899-8e50-1b55db8a4b1b-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-2w7h6\" (UID: \"1aa3545f-2866-4899-8e50-1b55db8a4b1b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-2w7h6" Nov 03 01:35:04.332920 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:04.332806 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aa3545f-2866-4899-8e50-1b55db8a4b1b-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-2w7h6\" (UID: \"1aa3545f-2866-4899-8e50-1b55db8a4b1b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-2w7h6" Nov 03 01:35:04.333103 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:04.333052 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1aa3545f-2866-4899-8e50-1b55db8a4b1b-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-2w7h6\" (UID: \"1aa3545f-2866-4899-8e50-1b55db8a4b1b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-2w7h6" Nov 03 01:35:04.333191 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:04.333168 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1aa3545f-2866-4899-8e50-1b55db8a4b1b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-2w7h6\" (UID: \"1aa3545f-2866-4899-8e50-1b55db8a4b1b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-2w7h6" Nov 03 01:35:04.333259 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:04.333234 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aa3545f-2866-4899-8e50-1b55db8a4b1b-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-2w7h6\" (UID: \"1aa3545f-2866-4899-8e50-1b55db8a4b1b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-2w7h6" Nov 03 01:35:04.334766 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:04.334739 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1aa3545f-2866-4899-8e50-1b55db8a4b1b-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-2w7h6\" (UID: \"1aa3545f-2866-4899-8e50-1b55db8a4b1b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-2w7h6" Nov 03 01:35:04.335150 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:04.335126 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1aa3545f-2866-4899-8e50-1b55db8a4b1b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-2w7h6\" (UID: \"1aa3545f-2866-4899-8e50-1b55db8a4b1b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-2w7h6" Nov 03 01:35:04.394972 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:04.394928 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1aa3545f-2866-4899-8e50-1b55db8a4b1b-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-2w7h6\" (UID: \"1aa3545f-2866-4899-8e50-1b55db8a4b1b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-2w7h6" Nov 03 01:35:04.395133 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:04.395038 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46n8n\" (UniqueName: \"kubernetes.io/projected/1aa3545f-2866-4899-8e50-1b55db8a4b1b-kube-api-access-46n8n\") pod \"maas-default-gateway-openshift-default-59f957fdd7-2w7h6\" (UID: \"1aa3545f-2866-4899-8e50-1b55db8a4b1b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-2w7h6" Nov 03 01:35:04.466348 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:04.466239 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-2w7h6" Nov 03 01:35:04.887590 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:35:04.887559 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1aa3545f_2866_4899_8e50_1b55db8a4b1b.slice/crio-b2e02594bfa12e4497dcc44f61266d015342aa4e7448ddfae82215cf38de9c5a WatchSource:0}: Error finding container b2e02594bfa12e4497dcc44f61266d015342aa4e7448ddfae82215cf38de9c5a: Status 404 returned error can't find the container with id b2e02594bfa12e4497dcc44f61266d015342aa4e7448ddfae82215cf38de9c5a Nov 03 01:35:04.890343 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:04.890306 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 01:35:04.890465 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:04.890417 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 01:35:04.890465 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:04.890452 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 01:35:04.890845 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:04.890823 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-2w7h6"] Nov 03 01:35:04.968271 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:04.968234 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-2w7h6" event={"ID":"1aa3545f-2866-4899-8e50-1b55db8a4b1b","Type":"ContainerStarted","Data":"b2e02594bfa12e4497dcc44f61266d015342aa4e7448ddfae82215cf38de9c5a"} Nov 03 01:35:05.976221 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:05.976178 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-2w7h6" event={"ID":"1aa3545f-2866-4899-8e50-1b55db8a4b1b","Type":"ContainerStarted","Data":"cdc63d9b0d92517e5191cddc28deafcd772da29fbbe48ca9d3db9c7c916099f1"} Nov 03 01:35:06.078265 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:06.078228 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:35:06.189771 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:06.189715 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-2w7h6" podStartSLOduration=2.189701453 podStartE2EDuration="2.189701453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:35:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:35:06.154769167 +0000 UTC m=+988.184251475" watchObservedRunningTime="2026-11-03 01:35:06.189701453 +0000 UTC m=+988.219183747" Nov 03 01:35:06.466830 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:06.466797 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-2w7h6" Nov 03 01:35:06.468562 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:06.468530 5437 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-2w7h6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 03 01:35:06.468694 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:06.468596 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-2w7h6" podUID="1aa3545f-2866-4899-8e50-1b55db8a4b1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 03 01:35:07.467211 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:07.467179 5437 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-2w7h6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 03 01:35:07.467657 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:07.467254 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-2w7h6" podUID="1aa3545f-2866-4899-8e50-1b55db8a4b1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 03 01:35:08.467192 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:08.467147 5437 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-2w7h6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 03 01:35:08.467357 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:08.467205 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-2w7h6" podUID="1aa3545f-2866-4899-8e50-1b55db8a4b1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 03 01:35:09.466962 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:09.466927 5437 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-2w7h6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 03 01:35:09.467172 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:09.466991 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-2w7h6" podUID="1aa3545f-2866-4899-8e50-1b55db8a4b1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 03 01:35:10.467629 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:10.467589 5437 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-2w7h6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 03 01:35:10.468068 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:10.467651 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-2w7h6" podUID="1aa3545f-2866-4899-8e50-1b55db8a4b1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 03 01:35:10.935399 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:10.926044 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:35:11.466918 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:11.466879 5437 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-2w7h6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 03 01:35:11.467111 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:11.466949 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-2w7h6" podUID="1aa3545f-2866-4899-8e50-1b55db8a4b1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 03 01:35:12.466949 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:12.466909 5437 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-2w7h6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 03 01:35:12.467471 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:12.466979 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-2w7h6" podUID="1aa3545f-2866-4899-8e50-1b55db8a4b1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 03 01:35:12.786655 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:12.786565 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-e1793d/e2e-conflict-a-9de9e4b3-kserve-79884b557b-bp2sj"] Nov 03 01:35:12.790671 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:12.790644 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e1793d/e2e-conflict-a-9de9e4b3-kserve-79884b557b-bp2sj" Nov 03 01:35:12.830696 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:12.830647 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a290bd4-9950-4806-bf86-1f02c8b60ea6-home\") pod \"e2e-conflict-a-9de9e4b3-kserve-79884b557b-bp2sj\" (UID: \"9a290bd4-9950-4806-bf86-1f02c8b60ea6\") " pod="e2e-models-e2e-worker-w1-e1793d/e2e-conflict-a-9de9e4b3-kserve-79884b557b-bp2sj" Nov 03 01:35:12.830893 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:12.830736 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-465d4\" (UniqueName: \"kubernetes.io/projected/9a290bd4-9950-4806-bf86-1f02c8b60ea6-kube-api-access-465d4\") pod \"e2e-conflict-a-9de9e4b3-kserve-79884b557b-bp2sj\" (UID: \"9a290bd4-9950-4806-bf86-1f02c8b60ea6\") " pod="e2e-models-e2e-worker-w1-e1793d/e2e-conflict-a-9de9e4b3-kserve-79884b557b-bp2sj" Nov 03 01:35:12.830893 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:12.830792 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a290bd4-9950-4806-bf86-1f02c8b60ea6-model-cache\") pod \"e2e-conflict-a-9de9e4b3-kserve-79884b557b-bp2sj\" (UID: \"9a290bd4-9950-4806-bf86-1f02c8b60ea6\") " pod="e2e-models-e2e-worker-w1-e1793d/e2e-conflict-a-9de9e4b3-kserve-79884b557b-bp2sj" Nov 03 01:35:12.830893 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:12.830887 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a290bd4-9950-4806-bf86-1f02c8b60ea6-tls-certs\") pod \"e2e-conflict-a-9de9e4b3-kserve-79884b557b-bp2sj\" (UID: \"9a290bd4-9950-4806-bf86-1f02c8b60ea6\") " pod="e2e-models-e2e-worker-w1-e1793d/e2e-conflict-a-9de9e4b3-kserve-79884b557b-bp2sj" Nov 03 01:35:12.831062 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:12.830943 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a290bd4-9950-4806-bf86-1f02c8b60ea6-tmp-dir\") pod \"e2e-conflict-a-9de9e4b3-kserve-79884b557b-bp2sj\" (UID: \"9a290bd4-9950-4806-bf86-1f02c8b60ea6\") " pod="e2e-models-e2e-worker-w1-e1793d/e2e-conflict-a-9de9e4b3-kserve-79884b557b-bp2sj" Nov 03 01:35:12.831062 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:12.830994 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a290bd4-9950-4806-bf86-1f02c8b60ea6-dshm\") pod \"e2e-conflict-a-9de9e4b3-kserve-79884b557b-bp2sj\" (UID: \"9a290bd4-9950-4806-bf86-1f02c8b60ea6\") " pod="e2e-models-e2e-worker-w1-e1793d/e2e-conflict-a-9de9e4b3-kserve-79884b557b-bp2sj" Nov 03 01:35:12.848010 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:12.847967 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-e1793d/e2e-conflict-a-9de9e4b3-kserve-79884b557b-bp2sj"] Nov 03 01:35:12.896416 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:12.896376 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-e1793d\"/\"e2e-conflict-a-9de9e4b3-kserve-self-signed-certs\"" Nov 03 01:35:12.932280 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:12.932233 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a290bd4-9950-4806-bf86-1f02c8b60ea6-dshm\") pod \"e2e-conflict-a-9de9e4b3-kserve-79884b557b-bp2sj\" (UID: \"9a290bd4-9950-4806-bf86-1f02c8b60ea6\") " pod="e2e-models-e2e-worker-w1-e1793d/e2e-conflict-a-9de9e4b3-kserve-79884b557b-bp2sj" Nov 03 01:35:12.932518 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:12.932304 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a290bd4-9950-4806-bf86-1f02c8b60ea6-home\") pod \"e2e-conflict-a-9de9e4b3-kserve-79884b557b-bp2sj\" (UID: \"9a290bd4-9950-4806-bf86-1f02c8b60ea6\") " pod="e2e-models-e2e-worker-w1-e1793d/e2e-conflict-a-9de9e4b3-kserve-79884b557b-bp2sj" Nov 03 01:35:12.932518 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:12.932399 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-465d4\" (UniqueName: \"kubernetes.io/projected/9a290bd4-9950-4806-bf86-1f02c8b60ea6-kube-api-access-465d4\") pod \"e2e-conflict-a-9de9e4b3-kserve-79884b557b-bp2sj\" (UID: \"9a290bd4-9950-4806-bf86-1f02c8b60ea6\") " pod="e2e-models-e2e-worker-w1-e1793d/e2e-conflict-a-9de9e4b3-kserve-79884b557b-bp2sj" Nov 03 01:35:12.932518 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:12.932461 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a290bd4-9950-4806-bf86-1f02c8b60ea6-model-cache\") pod \"e2e-conflict-a-9de9e4b3-kserve-79884b557b-bp2sj\" (UID: \"9a290bd4-9950-4806-bf86-1f02c8b60ea6\") " pod="e2e-models-e2e-worker-w1-e1793d/e2e-conflict-a-9de9e4b3-kserve-79884b557b-bp2sj" Nov 03 01:35:12.932712 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:12.932576 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a290bd4-9950-4806-bf86-1f02c8b60ea6-tls-certs\") pod \"e2e-conflict-a-9de9e4b3-kserve-79884b557b-bp2sj\" (UID: \"9a290bd4-9950-4806-bf86-1f02c8b60ea6\") " pod="e2e-models-e2e-worker-w1-e1793d/e2e-conflict-a-9de9e4b3-kserve-79884b557b-bp2sj" Nov 03 01:35:12.932712 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:12.932630 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a290bd4-9950-4806-bf86-1f02c8b60ea6-tmp-dir\") pod \"e2e-conflict-a-9de9e4b3-kserve-79884b557b-bp2sj\" (UID: \"9a290bd4-9950-4806-bf86-1f02c8b60ea6\") " pod="e2e-models-e2e-worker-w1-e1793d/e2e-conflict-a-9de9e4b3-kserve-79884b557b-bp2sj" Nov 03 01:35:12.932893 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:12.932865 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a290bd4-9950-4806-bf86-1f02c8b60ea6-home\") pod \"e2e-conflict-a-9de9e4b3-kserve-79884b557b-bp2sj\" (UID: \"9a290bd4-9950-4806-bf86-1f02c8b60ea6\") " pod="e2e-models-e2e-worker-w1-e1793d/e2e-conflict-a-9de9e4b3-kserve-79884b557b-bp2sj" Nov 03 01:35:12.932962 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:12.932912 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a290bd4-9950-4806-bf86-1f02c8b60ea6-model-cache\") pod \"e2e-conflict-a-9de9e4b3-kserve-79884b557b-bp2sj\" (UID: \"9a290bd4-9950-4806-bf86-1f02c8b60ea6\") " pod="e2e-models-e2e-worker-w1-e1793d/e2e-conflict-a-9de9e4b3-kserve-79884b557b-bp2sj" Nov 03 01:35:12.933065 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:12.933033 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a290bd4-9950-4806-bf86-1f02c8b60ea6-tmp-dir\") pod \"e2e-conflict-a-9de9e4b3-kserve-79884b557b-bp2sj\" (UID: \"9a290bd4-9950-4806-bf86-1f02c8b60ea6\") " pod="e2e-models-e2e-worker-w1-e1793d/e2e-conflict-a-9de9e4b3-kserve-79884b557b-bp2sj" Nov 03 01:35:12.935255 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:12.935225 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a290bd4-9950-4806-bf86-1f02c8b60ea6-tls-certs\") pod \"e2e-conflict-a-9de9e4b3-kserve-79884b557b-bp2sj\" (UID: \"9a290bd4-9950-4806-bf86-1f02c8b60ea6\") " pod="e2e-models-e2e-worker-w1-e1793d/e2e-conflict-a-9de9e4b3-kserve-79884b557b-bp2sj" Nov 03 01:35:12.935446 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:12.935403 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a290bd4-9950-4806-bf86-1f02c8b60ea6-dshm\") pod \"e2e-conflict-a-9de9e4b3-kserve-79884b557b-bp2sj\" (UID: \"9a290bd4-9950-4806-bf86-1f02c8b60ea6\") " pod="e2e-models-e2e-worker-w1-e1793d/e2e-conflict-a-9de9e4b3-kserve-79884b557b-bp2sj" Nov 03 01:35:12.947913 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:12.947875 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:35:13.007126 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:13.007084 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-465d4\" (UniqueName: \"kubernetes.io/projected/9a290bd4-9950-4806-bf86-1f02c8b60ea6-kube-api-access-465d4\") pod \"e2e-conflict-a-9de9e4b3-kserve-79884b557b-bp2sj\" (UID: \"9a290bd4-9950-4806-bf86-1f02c8b60ea6\") " pod="e2e-models-e2e-worker-w1-e1793d/e2e-conflict-a-9de9e4b3-kserve-79884b557b-bp2sj" Nov 03 01:35:13.104656 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:13.104609 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e1793d/e2e-conflict-a-9de9e4b3-kserve-79884b557b-bp2sj" Nov 03 01:35:13.497782 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:35:13.497746 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9a290bd4_9950_4806_bf86_1f02c8b60ea6.slice/crio-ef0d62f4717b950b63c3fb0c824e398548e882a424f0b8ef87eab1b2002ca68c WatchSource:0}: Error finding container ef0d62f4717b950b63c3fb0c824e398548e882a424f0b8ef87eab1b2002ca68c: Status 404 returned error can't find the container with id ef0d62f4717b950b63c3fb0c824e398548e882a424f0b8ef87eab1b2002ca68c Nov 03 01:35:13.518841 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:13.518808 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-e1793d/e2e-conflict-a-9de9e4b3-kserve-79884b557b-bp2sj"] Nov 03 01:35:13.831985 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:13.831949 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-2w7h6" Nov 03 01:35:13.832292 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:13.832264 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-2w7h6" Nov 03 01:35:13.833082 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:13.833059 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-2w7h6" Nov 03 01:35:14.037034 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:35:14.037001 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 01:35:14.038035 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:14.037750 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e1793d/e2e-conflict-a-9de9e4b3-kserve-79884b557b-bp2sj" event={"ID":"9a290bd4-9950-4806-bf86-1f02c8b60ea6","Type":"ContainerStarted","Data":"e34f717cfe4652199ac2027d0d735950b648d20662872f890f6818171dfb3ac0"} Nov 03 01:35:14.038035 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:14.037793 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e1793d/e2e-conflict-a-9de9e4b3-kserve-79884b557b-bp2sj" event={"ID":"9a290bd4-9950-4806-bf86-1f02c8b60ea6","Type":"ContainerStarted","Data":"ef0d62f4717b950b63c3fb0c824e398548e882a424f0b8ef87eab1b2002ca68c"} Nov 03 01:35:14.105394 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:14.105269 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e1793d/e2e-conflict-a-9de9e4b3-kserve-79884b557b-bp2sj" Nov 03 01:35:14.202823 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:14.202767 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-e1793d/e2e-conflict-a-9de9e4b3-kserve-79884b557b-bp2sj" podStartSLOduration=2.202750754 podStartE2EDuration="2.202750754s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:35:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:35:14.200330684 +0000 UTC m=+996.229812980" watchObservedRunningTime="2026-11-03 01:35:14.202750754 +0000 UTC m=+996.232233078" Nov 03 01:35:15.984593 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:15.984557 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:35:20.795661 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:20.795546 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:35:23.105176 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:23.105137 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-e1793d/e2e-conflict-a-9de9e4b3-kserve-79884b557b-bp2sj" Nov 03 01:35:23.117737 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:23.117702 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-e1793d/e2e-conflict-a-9de9e4b3-kserve-79884b557b-bp2sj" Nov 03 01:35:24.105764 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:24.105729 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e1793d/e2e-conflict-a-9de9e4b3-kserve-79884b557b-bp2sj" Nov 03 01:35:25.575099 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:25.575059 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:35:30.076916 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:30.076876 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-2tzm2"] Nov 03 01:35:30.080705 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:30.080687 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-2tzm2" Nov 03 01:35:30.135191 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:30.135155 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-9da431-openshift-default-dockercfg-bq94q\"" Nov 03 01:35:30.189102 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:30.189065 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-2tzm2"] Nov 03 01:35:30.230167 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:30.230128 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1006e22-dda3-4a64-b5c2-a711362b032d-istio-data\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-2tzm2\" (UID: \"f1006e22-dda3-4a64-b5c2-a711362b032d\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-2tzm2" Nov 03 01:35:30.230356 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:30.230174 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1006e22-dda3-4a64-b5c2-a711362b032d-workload-certs\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-2tzm2\" (UID: \"f1006e22-dda3-4a64-b5c2-a711362b032d\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-2tzm2" Nov 03 01:35:30.230356 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:30.230324 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1006e22-dda3-4a64-b5c2-a711362b032d-workload-socket\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-2tzm2\" (UID: \"f1006e22-dda3-4a64-b5c2-a711362b032d\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-2tzm2" Nov 03 01:35:30.230356 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:30.230349 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1006e22-dda3-4a64-b5c2-a711362b032d-istio-token\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-2tzm2\" (UID: \"f1006e22-dda3-4a64-b5c2-a711362b032d\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-2tzm2" Nov 03 01:35:30.230523 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:30.230508 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1006e22-dda3-4a64-b5c2-a711362b032d-credential-socket\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-2tzm2\" (UID: \"f1006e22-dda3-4a64-b5c2-a711362b032d\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-2tzm2" Nov 03 01:35:30.230573 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:30.230557 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1006e22-dda3-4a64-b5c2-a711362b032d-istio-envoy\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-2tzm2\" (UID: \"f1006e22-dda3-4a64-b5c2-a711362b032d\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-2tzm2" Nov 03 01:35:30.230634 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:30.230608 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1006e22-dda3-4a64-b5c2-a711362b032d-istio-podinfo\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-2tzm2\" (UID: \"f1006e22-dda3-4a64-b5c2-a711362b032d\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-2tzm2" Nov 03 01:35:30.230702 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:30.230635 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1006e22-dda3-4a64-b5c2-a711362b032d-istiod-ca-cert\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-2tzm2\" (UID: \"f1006e22-dda3-4a64-b5c2-a711362b032d\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-2tzm2" Nov 03 01:35:30.230702 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:30.230660 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqrgx\" (UniqueName: \"kubernetes.io/projected/f1006e22-dda3-4a64-b5c2-a711362b032d-kube-api-access-dqrgx\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-2tzm2\" (UID: \"f1006e22-dda3-4a64-b5c2-a711362b032d\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-2tzm2" Nov 03 01:35:30.332441 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:30.332304 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1006e22-dda3-4a64-b5c2-a711362b032d-workload-socket\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-2tzm2\" (UID: \"f1006e22-dda3-4a64-b5c2-a711362b032d\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-2tzm2" Nov 03 01:35:30.332441 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:30.332354 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1006e22-dda3-4a64-b5c2-a711362b032d-istio-token\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-2tzm2\" (UID: \"f1006e22-dda3-4a64-b5c2-a711362b032d\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-2tzm2" Nov 03 01:35:30.332441 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:30.332426 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1006e22-dda3-4a64-b5c2-a711362b032d-credential-socket\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-2tzm2\" (UID: \"f1006e22-dda3-4a64-b5c2-a711362b032d\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-2tzm2" Nov 03 01:35:30.332737 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:30.332454 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1006e22-dda3-4a64-b5c2-a711362b032d-istio-envoy\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-2tzm2\" (UID: \"f1006e22-dda3-4a64-b5c2-a711362b032d\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-2tzm2" Nov 03 01:35:30.332737 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:30.332473 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1006e22-dda3-4a64-b5c2-a711362b032d-istio-podinfo\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-2tzm2\" (UID: \"f1006e22-dda3-4a64-b5c2-a711362b032d\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-2tzm2" Nov 03 01:35:30.332737 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:30.332565 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1006e22-dda3-4a64-b5c2-a711362b032d-istiod-ca-cert\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-2tzm2\" (UID: \"f1006e22-dda3-4a64-b5c2-a711362b032d\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-2tzm2" Nov 03 01:35:30.332737 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:30.332615 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqrgx\" (UniqueName: \"kubernetes.io/projected/f1006e22-dda3-4a64-b5c2-a711362b032d-kube-api-access-dqrgx\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-2tzm2\" (UID: \"f1006e22-dda3-4a64-b5c2-a711362b032d\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-2tzm2" Nov 03 01:35:30.332737 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:30.332674 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1006e22-dda3-4a64-b5c2-a711362b032d-istio-data\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-2tzm2\" (UID: \"f1006e22-dda3-4a64-b5c2-a711362b032d\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-2tzm2" Nov 03 01:35:30.332737 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:30.332713 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1006e22-dda3-4a64-b5c2-a711362b032d-workload-certs\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-2tzm2\" (UID: \"f1006e22-dda3-4a64-b5c2-a711362b032d\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-2tzm2" Nov 03 01:35:30.333045 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:30.332838 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1006e22-dda3-4a64-b5c2-a711362b032d-workload-socket\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-2tzm2\" (UID: \"f1006e22-dda3-4a64-b5c2-a711362b032d\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-2tzm2" Nov 03 01:35:30.333045 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:30.332890 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1006e22-dda3-4a64-b5c2-a711362b032d-credential-socket\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-2tzm2\" (UID: \"f1006e22-dda3-4a64-b5c2-a711362b032d\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-2tzm2" Nov 03 01:35:30.333175 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:30.333153 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1006e22-dda3-4a64-b5c2-a711362b032d-istio-data\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-2tzm2\" (UID: \"f1006e22-dda3-4a64-b5c2-a711362b032d\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-2tzm2" Nov 03 01:35:30.333254 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:30.333231 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1006e22-dda3-4a64-b5c2-a711362b032d-workload-certs\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-2tzm2\" (UID: \"f1006e22-dda3-4a64-b5c2-a711362b032d\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-2tzm2" Nov 03 01:35:30.333414 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:30.333276 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1006e22-dda3-4a64-b5c2-a711362b032d-istiod-ca-cert\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-2tzm2\" (UID: \"f1006e22-dda3-4a64-b5c2-a711362b032d\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-2tzm2" Nov 03 01:35:30.334940 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:30.334915 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1006e22-dda3-4a64-b5c2-a711362b032d-istio-envoy\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-2tzm2\" (UID: \"f1006e22-dda3-4a64-b5c2-a711362b032d\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-2tzm2" Nov 03 01:35:30.335036 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:30.335014 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1006e22-dda3-4a64-b5c2-a711362b032d-istio-podinfo\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-2tzm2\" (UID: \"f1006e22-dda3-4a64-b5c2-a711362b032d\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-2tzm2" Nov 03 01:35:30.376980 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:30.376938 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:35:30.395549 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:30.395511 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1006e22-dda3-4a64-b5c2-a711362b032d-istio-token\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-2tzm2\" (UID: \"f1006e22-dda3-4a64-b5c2-a711362b032d\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-2tzm2" Nov 03 01:35:30.395713 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:30.395621 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqrgx\" (UniqueName: \"kubernetes.io/projected/f1006e22-dda3-4a64-b5c2-a711362b032d-kube-api-access-dqrgx\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-2tzm2\" (UID: \"f1006e22-dda3-4a64-b5c2-a711362b032d\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-2tzm2" Nov 03 01:35:30.396255 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:30.396235 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-2tzm2" Nov 03 01:35:30.818696 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:35:30.818662 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1006e22_dda3_4a64_b5c2_a711362b032d.slice/crio-b3f563adacdac3417659eb40675e10de87f637e9ce3591c2bad9f459e4fc0399 WatchSource:0}: Error finding container b3f563adacdac3417659eb40675e10de87f637e9ce3591c2bad9f459e4fc0399: Status 404 returned error can't find the container with id b3f563adacdac3417659eb40675e10de87f637e9ce3591c2bad9f459e4fc0399 Nov 03 01:35:30.821178 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:30.821144 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 01:35:30.821263 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:30.821215 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 01:35:30.821263 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:30.821243 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 01:35:30.821614 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:30.821590 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-2tzm2"] Nov 03 01:35:31.133964 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:31.133867 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-2tzm2" event={"ID":"f1006e22-dda3-4a64-b5c2-a711362b032d","Type":"ContainerStarted","Data":"bfae63d3f41e34e4b7044ec53e391b5dc6a26d5b0e7a39ff82029de31d82b064"} Nov 03 01:35:31.133964 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:31.133925 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-2tzm2" event={"ID":"f1006e22-dda3-4a64-b5c2-a711362b032d","Type":"ContainerStarted","Data":"b3f563adacdac3417659eb40675e10de87f637e9ce3591c2bad9f459e4fc0399"} Nov 03 01:35:31.257764 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:31.257709 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-2tzm2" podStartSLOduration=1.257694747 podStartE2EDuration="1.257694747s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:35:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:35:31.257203308 +0000 UTC m=+1013.286685608" watchObservedRunningTime="2026-11-03 01:35:31.257694747 +0000 UTC m=+1013.287177042" Nov 03 01:35:31.396693 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:31.396594 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-2tzm2" Nov 03 01:35:31.398229 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:31.398198 5437 patch_prober.go:29] interesting pod/e2e-worker-w2-9da431-openshift-default-65c799fd77-2tzm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 03 01:35:31.398382 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:31.398249 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-2tzm2" podUID="f1006e22-dda3-4a64-b5c2-a711362b032d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 03 01:35:32.396978 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:32.396940 5437 patch_prober.go:29] interesting pod/e2e-worker-w2-9da431-openshift-default-65c799fd77-2tzm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 03 01:35:32.397445 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:32.397007 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-2tzm2" podUID="f1006e22-dda3-4a64-b5c2-a711362b032d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 03 01:35:33.398211 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:33.398175 5437 patch_prober.go:29] interesting pod/e2e-worker-w2-9da431-openshift-default-65c799fd77-2tzm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 03 01:35:33.398721 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:33.398233 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-2tzm2" podUID="f1006e22-dda3-4a64-b5c2-a711362b032d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 03 01:35:34.397340 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:34.397304 5437 patch_prober.go:29] interesting pod/e2e-worker-w2-9da431-openshift-default-65c799fd77-2tzm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 03 01:35:34.397558 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:34.397394 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-2tzm2" podUID="f1006e22-dda3-4a64-b5c2-a711362b032d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 03 01:35:35.194830 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:35.194790 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:35:35.397314 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:35.397278 5437 patch_prober.go:29] interesting pod/e2e-worker-w2-9da431-openshift-default-65c799fd77-2tzm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 03 01:35:35.397531 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:35.397344 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-2tzm2" podUID="f1006e22-dda3-4a64-b5c2-a711362b032d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 03 01:35:36.397040 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:36.397000 5437 patch_prober.go:29] interesting pod/e2e-worker-w2-9da431-openshift-default-65c799fd77-2tzm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 03 01:35:36.397483 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:36.397071 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-2tzm2" podUID="f1006e22-dda3-4a64-b5c2-a711362b032d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 03 01:35:37.602246 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:37.602211 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-2tzm2" Nov 03 01:35:37.602763 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:37.602647 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-2tzm2" Nov 03 01:35:37.603454 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:37.603431 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-2tzm2" Nov 03 01:35:39.988112 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:39.988068 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:35:41.359083 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:41.359045 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:35:44.034554 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:35:44.034514 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 01:35:44.730736 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:44.730692 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:35:46.391702 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:46.391661 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hhnxh"] Nov 03 01:35:46.398202 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:46.398158 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hhnxh" Nov 03 01:35:46.461830 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:46.461794 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-e1793d-openshift-default-dockercfg-4jbqs\"" Nov 03 01:35:46.508895 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:46.508853 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/978393a8-09d7-48b6-9104-d1cae96b8f18-workload-certs\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hhnxh\" (UID: \"978393a8-09d7-48b6-9104-d1cae96b8f18\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hhnxh" Nov 03 01:35:46.509085 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:46.508934 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/978393a8-09d7-48b6-9104-d1cae96b8f18-credential-socket\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hhnxh\" (UID: \"978393a8-09d7-48b6-9104-d1cae96b8f18\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hhnxh" Nov 03 01:35:46.509085 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:46.508964 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/978393a8-09d7-48b6-9104-d1cae96b8f18-istio-data\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hhnxh\" (UID: \"978393a8-09d7-48b6-9104-d1cae96b8f18\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hhnxh" Nov 03 01:35:46.509085 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:46.509010 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/978393a8-09d7-48b6-9104-d1cae96b8f18-istio-token\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hhnxh\" (UID: \"978393a8-09d7-48b6-9104-d1cae96b8f18\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hhnxh" Nov 03 01:35:46.509238 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:46.509108 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/978393a8-09d7-48b6-9104-d1cae96b8f18-workload-socket\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hhnxh\" (UID: \"978393a8-09d7-48b6-9104-d1cae96b8f18\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hhnxh" Nov 03 01:35:46.509289 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:46.509241 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxtmb\" (UniqueName: \"kubernetes.io/projected/978393a8-09d7-48b6-9104-d1cae96b8f18-kube-api-access-jxtmb\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hhnxh\" (UID: \"978393a8-09d7-48b6-9104-d1cae96b8f18\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hhnxh" Nov 03 01:35:46.509289 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:46.509281 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/978393a8-09d7-48b6-9104-d1cae96b8f18-istiod-ca-cert\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hhnxh\" (UID: \"978393a8-09d7-48b6-9104-d1cae96b8f18\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hhnxh" Nov 03 01:35:46.509448 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:46.509413 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/978393a8-09d7-48b6-9104-d1cae96b8f18-istio-podinfo\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hhnxh\" (UID: \"978393a8-09d7-48b6-9104-d1cae96b8f18\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hhnxh" Nov 03 01:35:46.509524 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:46.509479 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/978393a8-09d7-48b6-9104-d1cae96b8f18-istio-envoy\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hhnxh\" (UID: \"978393a8-09d7-48b6-9104-d1cae96b8f18\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hhnxh" Nov 03 01:35:46.515693 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:46.515668 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hhnxh"] Nov 03 01:35:46.612889 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:46.610917 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxtmb\" (UniqueName: \"kubernetes.io/projected/978393a8-09d7-48b6-9104-d1cae96b8f18-kube-api-access-jxtmb\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hhnxh\" (UID: \"978393a8-09d7-48b6-9104-d1cae96b8f18\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hhnxh" Nov 03 01:35:46.612889 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:46.610963 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/978393a8-09d7-48b6-9104-d1cae96b8f18-istiod-ca-cert\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hhnxh\" (UID: \"978393a8-09d7-48b6-9104-d1cae96b8f18\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hhnxh" Nov 03 01:35:46.612889 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:46.611007 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/978393a8-09d7-48b6-9104-d1cae96b8f18-istio-podinfo\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hhnxh\" (UID: \"978393a8-09d7-48b6-9104-d1cae96b8f18\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hhnxh" Nov 03 01:35:46.612889 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:46.611042 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/978393a8-09d7-48b6-9104-d1cae96b8f18-istio-envoy\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hhnxh\" (UID: \"978393a8-09d7-48b6-9104-d1cae96b8f18\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hhnxh" Nov 03 01:35:46.612889 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:46.611084 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/978393a8-09d7-48b6-9104-d1cae96b8f18-workload-certs\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hhnxh\" (UID: \"978393a8-09d7-48b6-9104-d1cae96b8f18\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hhnxh" Nov 03 01:35:46.612889 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:46.611103 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/978393a8-09d7-48b6-9104-d1cae96b8f18-credential-socket\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hhnxh\" (UID: \"978393a8-09d7-48b6-9104-d1cae96b8f18\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hhnxh" Nov 03 01:35:46.612889 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:46.611119 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/978393a8-09d7-48b6-9104-d1cae96b8f18-istio-data\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hhnxh\" (UID: \"978393a8-09d7-48b6-9104-d1cae96b8f18\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hhnxh" Nov 03 01:35:46.612889 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:46.611137 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/978393a8-09d7-48b6-9104-d1cae96b8f18-istio-token\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hhnxh\" (UID: \"978393a8-09d7-48b6-9104-d1cae96b8f18\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hhnxh" Nov 03 01:35:46.612889 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:46.611159 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/978393a8-09d7-48b6-9104-d1cae96b8f18-workload-socket\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hhnxh\" (UID: \"978393a8-09d7-48b6-9104-d1cae96b8f18\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hhnxh" Nov 03 01:35:46.612889 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:46.611572 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/978393a8-09d7-48b6-9104-d1cae96b8f18-workload-socket\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hhnxh\" (UID: \"978393a8-09d7-48b6-9104-d1cae96b8f18\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hhnxh" Nov 03 01:35:46.612889 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:46.612253 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/978393a8-09d7-48b6-9104-d1cae96b8f18-istiod-ca-cert\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hhnxh\" (UID: \"978393a8-09d7-48b6-9104-d1cae96b8f18\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hhnxh" Nov 03 01:35:46.613661 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:46.613274 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/978393a8-09d7-48b6-9104-d1cae96b8f18-workload-certs\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hhnxh\" (UID: \"978393a8-09d7-48b6-9104-d1cae96b8f18\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hhnxh" Nov 03 01:35:46.613661 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:46.613466 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/978393a8-09d7-48b6-9104-d1cae96b8f18-istio-data\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hhnxh\" (UID: \"978393a8-09d7-48b6-9104-d1cae96b8f18\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hhnxh" Nov 03 01:35:46.613661 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:46.613561 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/978393a8-09d7-48b6-9104-d1cae96b8f18-credential-socket\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hhnxh\" (UID: \"978393a8-09d7-48b6-9104-d1cae96b8f18\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hhnxh" Nov 03 01:35:46.615679 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:46.615651 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/978393a8-09d7-48b6-9104-d1cae96b8f18-istio-podinfo\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hhnxh\" (UID: \"978393a8-09d7-48b6-9104-d1cae96b8f18\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hhnxh" Nov 03 01:35:46.615872 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:46.615853 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/978393a8-09d7-48b6-9104-d1cae96b8f18-istio-envoy\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hhnxh\" (UID: \"978393a8-09d7-48b6-9104-d1cae96b8f18\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hhnxh" Nov 03 01:35:46.695091 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:46.695012 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/978393a8-09d7-48b6-9104-d1cae96b8f18-istio-token\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hhnxh\" (UID: \"978393a8-09d7-48b6-9104-d1cae96b8f18\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hhnxh" Nov 03 01:35:46.701324 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:46.701258 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxtmb\" (UniqueName: \"kubernetes.io/projected/978393a8-09d7-48b6-9104-d1cae96b8f18-kube-api-access-jxtmb\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hhnxh\" (UID: \"978393a8-09d7-48b6-9104-d1cae96b8f18\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hhnxh" Nov 03 01:35:46.714580 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:46.714513 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hhnxh" Nov 03 01:35:47.293608 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:47.293559 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hhnxh"] Nov 03 01:35:47.301678 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:47.301643 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 01:35:47.301784 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:47.301710 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 01:35:47.301784 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:47.301739 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 01:35:48.305205 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:48.305169 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hhnxh" event={"ID":"978393a8-09d7-48b6-9104-d1cae96b8f18","Type":"ContainerStarted","Data":"e7532f72870ccc430bd769ccf699ef97fc9d6165591283ddfa39a1d721cbd687"} Nov 03 01:35:48.305723 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:48.305223 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hhnxh" event={"ID":"978393a8-09d7-48b6-9104-d1cae96b8f18","Type":"ContainerStarted","Data":"ca4fe70661f4d83d04bc51be3877b133d9a1be4c04b7cd3131b34e40cd49f0d4"} Nov 03 01:35:48.482875 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:48.482810 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hhnxh" podStartSLOduration=2.482794981 podStartE2EDuration="2.482794981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:35:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:35:48.48051858 +0000 UTC m=+1030.510000875" watchObservedRunningTime="2026-11-03 01:35:48.482794981 +0000 UTC m=+1030.512277275" Nov 03 01:35:48.714938 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:48.714854 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hhnxh" Nov 03 01:35:48.716442 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:48.716397 5437 patch_prober.go:29] interesting pod/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hhnxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 03 01:35:48.716576 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:48.716465 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hhnxh" podUID="978393a8-09d7-48b6-9104-d1cae96b8f18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 03 01:35:49.698737 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:49.698697 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:35:49.714911 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:49.714876 5437 patch_prober.go:29] interesting pod/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hhnxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 03 01:35:49.715053 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:49.714942 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hhnxh" podUID="978393a8-09d7-48b6-9104-d1cae96b8f18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 03 01:35:50.715905 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:50.715865 5437 patch_prober.go:29] interesting pod/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hhnxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 03 01:35:50.716311 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:50.715934 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hhnxh" podUID="978393a8-09d7-48b6-9104-d1cae96b8f18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 03 01:35:50.984103 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:50.984012 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:35:51.186986 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:51.186945 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:35:51.387841 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:51.387806 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:35:51.715627 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:51.715534 5437 patch_prober.go:29] interesting pod/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hhnxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 03 01:35:51.715627 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:51.715608 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hhnxh" podUID="978393a8-09d7-48b6-9104-d1cae96b8f18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 03 01:35:52.715555 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:52.715515 5437 patch_prober.go:29] interesting pod/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hhnxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 03 01:35:52.715962 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:52.715577 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hhnxh" podUID="978393a8-09d7-48b6-9104-d1cae96b8f18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 03 01:35:53.715123 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:53.715085 5437 patch_prober.go:29] interesting pod/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hhnxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 03 01:35:53.715315 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:53.715160 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hhnxh" podUID="978393a8-09d7-48b6-9104-d1cae96b8f18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 03 01:35:54.513494 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:54.513458 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:35:54.719966 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:54.719934 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hhnxh" Nov 03 01:35:54.720385 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:54.720348 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hhnxh" Nov 03 01:35:54.721135 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:54.721116 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hhnxh" Nov 03 01:35:57.983143 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:57.983107 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:35:59.495477 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:35:59.495411 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:36:04.370194 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:36:04.370155 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:36:05.385112 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:36:05.385076 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:36:09.177612 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:36:09.177576 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:36:13.984347 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:36:13.984298 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:36:14.034214 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:36:14.034166 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 01:36:18.885394 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:36:18.885338 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:36:23.678857 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:36:23.678815 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:36:25.254145 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:36:25.254100 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:36:28.507180 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:36:28.507127 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:36:33.284202 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:36:33.284144 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:36:38.079865 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:36:38.079829 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:36:38.263469 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:36:38.263433 5437 scope.go:122] "RemoveContainer" containerID="693703fec71b62049ac6a7eda3bddf1624d98b1b543bb7a64f44ecc4edac6670" Nov 03 01:36:38.784039 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:36:38.783999 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-hj9f6"] Nov 03 01:36:38.789812 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:36:38.789783 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-hj9f6" Nov 03 01:36:38.844514 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:36:38.844479 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-b6fb9b-openshift-default-dockercfg-997hh\"" Nov 03 01:36:38.856031 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:36:38.856004 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b6b29be-71cc-442f-b430-0cc2344d400d-workload-certs\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-hj9f6\" (UID: \"6b6b29be-71cc-442f-b430-0cc2344d400d\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-hj9f6" Nov 03 01:36:38.856137 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:36:38.856037 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b6b29be-71cc-442f-b430-0cc2344d400d-istio-envoy\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-hj9f6\" (UID: \"6b6b29be-71cc-442f-b430-0cc2344d400d\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-hj9f6" Nov 03 01:36:38.856137 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:36:38.856060 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b6b29be-71cc-442f-b430-0cc2344d400d-istio-token\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-hj9f6\" (UID: \"6b6b29be-71cc-442f-b430-0cc2344d400d\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-hj9f6" Nov 03 01:36:38.856137 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:36:38.856077 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b6b29be-71cc-442f-b430-0cc2344d400d-workload-socket\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-hj9f6\" (UID: \"6b6b29be-71cc-442f-b430-0cc2344d400d\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-hj9f6" Nov 03 01:36:38.856252 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:36:38.856197 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqwrq\" (UniqueName: \"kubernetes.io/projected/6b6b29be-71cc-442f-b430-0cc2344d400d-kube-api-access-vqwrq\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-hj9f6\" (UID: \"6b6b29be-71cc-442f-b430-0cc2344d400d\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-hj9f6" Nov 03 01:36:38.856307 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:36:38.856293 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b6b29be-71cc-442f-b430-0cc2344d400d-istio-podinfo\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-hj9f6\" (UID: \"6b6b29be-71cc-442f-b430-0cc2344d400d\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-hj9f6" Nov 03 01:36:38.856357 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:36:38.856346 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b6b29be-71cc-442f-b430-0cc2344d400d-istio-data\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-hj9f6\" (UID: \"6b6b29be-71cc-442f-b430-0cc2344d400d\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-hj9f6" Nov 03 01:36:38.856424 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:36:38.856393 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b6b29be-71cc-442f-b430-0cc2344d400d-istiod-ca-cert\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-hj9f6\" (UID: \"6b6b29be-71cc-442f-b430-0cc2344d400d\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-hj9f6" Nov 03 01:36:38.856489 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:36:38.856476 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b6b29be-71cc-442f-b430-0cc2344d400d-credential-socket\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-hj9f6\" (UID: \"6b6b29be-71cc-442f-b430-0cc2344d400d\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-hj9f6" Nov 03 01:36:38.898522 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:36:38.898490 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-hj9f6"] Nov 03 01:36:38.958224 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:36:38.958182 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b6b29be-71cc-442f-b430-0cc2344d400d-workload-certs\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-hj9f6\" (UID: \"6b6b29be-71cc-442f-b430-0cc2344d400d\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-hj9f6" Nov 03 01:36:38.958428 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:36:38.958231 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b6b29be-71cc-442f-b430-0cc2344d400d-istio-envoy\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-hj9f6\" (UID: \"6b6b29be-71cc-442f-b430-0cc2344d400d\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-hj9f6" Nov 03 01:36:38.958428 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:36:38.958264 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b6b29be-71cc-442f-b430-0cc2344d400d-istio-token\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-hj9f6\" (UID: \"6b6b29be-71cc-442f-b430-0cc2344d400d\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-hj9f6" Nov 03 01:36:38.958428 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:36:38.958290 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b6b29be-71cc-442f-b430-0cc2344d400d-workload-socket\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-hj9f6\" (UID: \"6b6b29be-71cc-442f-b430-0cc2344d400d\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-hj9f6" Nov 03 01:36:38.958428 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:36:38.958319 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqwrq\" (UniqueName: \"kubernetes.io/projected/6b6b29be-71cc-442f-b430-0cc2344d400d-kube-api-access-vqwrq\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-hj9f6\" (UID: \"6b6b29be-71cc-442f-b430-0cc2344d400d\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-hj9f6" Nov 03 01:36:38.958428 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:36:38.958383 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b6b29be-71cc-442f-b430-0cc2344d400d-istio-podinfo\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-hj9f6\" (UID: \"6b6b29be-71cc-442f-b430-0cc2344d400d\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-hj9f6" Nov 03 01:36:38.958690 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:36:38.958454 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b6b29be-71cc-442f-b430-0cc2344d400d-istio-data\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-hj9f6\" (UID: \"6b6b29be-71cc-442f-b430-0cc2344d400d\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-hj9f6" Nov 03 01:36:38.958690 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:36:38.958481 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b6b29be-71cc-442f-b430-0cc2344d400d-istiod-ca-cert\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-hj9f6\" (UID: \"6b6b29be-71cc-442f-b430-0cc2344d400d\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-hj9f6" Nov 03 01:36:38.958690 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:36:38.958568 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b6b29be-71cc-442f-b430-0cc2344d400d-credential-socket\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-hj9f6\" (UID: \"6b6b29be-71cc-442f-b430-0cc2344d400d\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-hj9f6" Nov 03 01:36:38.958690 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:36:38.958667 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b6b29be-71cc-442f-b430-0cc2344d400d-workload-certs\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-hj9f6\" (UID: \"6b6b29be-71cc-442f-b430-0cc2344d400d\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-hj9f6" Nov 03 01:36:38.958896 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:36:38.958740 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b6b29be-71cc-442f-b430-0cc2344d400d-workload-socket\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-hj9f6\" (UID: \"6b6b29be-71cc-442f-b430-0cc2344d400d\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-hj9f6" Nov 03 01:36:38.958951 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:36:38.958923 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b6b29be-71cc-442f-b430-0cc2344d400d-credential-socket\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-hj9f6\" (UID: \"6b6b29be-71cc-442f-b430-0cc2344d400d\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-hj9f6" Nov 03 01:36:38.959005 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:36:38.958955 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b6b29be-71cc-442f-b430-0cc2344d400d-istio-data\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-hj9f6\" (UID: \"6b6b29be-71cc-442f-b430-0cc2344d400d\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-hj9f6" Nov 03 01:36:38.959502 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:36:38.959471 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b6b29be-71cc-442f-b430-0cc2344d400d-istiod-ca-cert\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-hj9f6\" (UID: \"6b6b29be-71cc-442f-b430-0cc2344d400d\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-hj9f6" Nov 03 01:36:38.961301 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:36:38.961271 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b6b29be-71cc-442f-b430-0cc2344d400d-istio-envoy\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-hj9f6\" (UID: \"6b6b29be-71cc-442f-b430-0cc2344d400d\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-hj9f6" Nov 03 01:36:38.961878 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:36:38.961857 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b6b29be-71cc-442f-b430-0cc2344d400d-istio-podinfo\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-hj9f6\" (UID: \"6b6b29be-71cc-442f-b430-0cc2344d400d\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-hj9f6" Nov 03 01:36:39.021575 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:36:39.021543 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b6b29be-71cc-442f-b430-0cc2344d400d-istio-token\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-hj9f6\" (UID: \"6b6b29be-71cc-442f-b430-0cc2344d400d\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-hj9f6" Nov 03 01:36:39.021778 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:36:39.021755 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqwrq\" (UniqueName: \"kubernetes.io/projected/6b6b29be-71cc-442f-b430-0cc2344d400d-kube-api-access-vqwrq\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-hj9f6\" (UID: \"6b6b29be-71cc-442f-b430-0cc2344d400d\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-hj9f6" Nov 03 01:36:39.103693 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:36:39.103660 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-hj9f6" Nov 03 01:36:39.526573 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:36:39.526519 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b6b29be_71cc_442f_b430_0cc2344d400d.slice/crio-58878f70fc646fd4a8b70417b5e938fbc42a47f565f5eb016fec5486c792d1bf WatchSource:0}: Error finding container 58878f70fc646fd4a8b70417b5e938fbc42a47f565f5eb016fec5486c792d1bf: Status 404 returned error can't find the container with id 58878f70fc646fd4a8b70417b5e938fbc42a47f565f5eb016fec5486c792d1bf Nov 03 01:36:39.529727 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:36:39.529699 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-hj9f6"] Nov 03 01:36:39.529838 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:36:39.529698 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 01:36:39.529838 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:36:39.529803 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 01:36:39.529938 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:36:39.529844 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 01:36:39.685150 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:36:39.685116 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-hj9f6" event={"ID":"6b6b29be-71cc-442f-b430-0cc2344d400d","Type":"ContainerStarted","Data":"9ae126f2a6a19e771de01ad82e4bfff14a7ed70738a87af663320230aea7117e"} Nov 03 01:36:39.685326 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:36:39.685171 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-hj9f6" event={"ID":"6b6b29be-71cc-442f-b430-0cc2344d400d","Type":"ContainerStarted","Data":"58878f70fc646fd4a8b70417b5e938fbc42a47f565f5eb016fec5486c792d1bf"} Nov 03 01:36:39.810111 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:36:39.810062 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-hj9f6" podStartSLOduration=1.8100492990000001 podStartE2EDuration="1.810049299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:36:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:36:39.809646682 +0000 UTC m=+1081.839128977" watchObservedRunningTime="2026-11-03 01:36:39.810049299 +0000 UTC m=+1081.839531595" Nov 03 01:36:40.104328 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:36:40.104249 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-hj9f6" Nov 03 01:36:40.106023 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:36:40.105993 5437 patch_prober.go:29] interesting pod/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-hj9f6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 03 01:36:40.106170 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:36:40.106039 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-hj9f6" podUID="6b6b29be-71cc-442f-b430-0cc2344d400d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 03 01:36:41.104083 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:36:41.104042 5437 patch_prober.go:29] interesting pod/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-hj9f6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 03 01:36:41.104250 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:36:41.104104 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-hj9f6" podUID="6b6b29be-71cc-442f-b430-0cc2344d400d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 03 01:36:41.700065 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:36:41.700033 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:36:42.104247 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:36:42.104210 5437 patch_prober.go:29] interesting pod/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-hj9f6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 03 01:36:42.104446 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:36:42.104278 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-hj9f6" podUID="6b6b29be-71cc-442f-b430-0cc2344d400d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 03 01:36:42.945594 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:36:42.945550 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:36:43.104015 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:36:43.103974 5437 patch_prober.go:29] interesting pod/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-hj9f6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 03 01:36:43.104196 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:36:43.104030 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-hj9f6" podUID="6b6b29be-71cc-442f-b430-0cc2344d400d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 03 01:36:44.055027 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:36:44.054990 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 01:36:44.104730 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:36:44.104692 5437 patch_prober.go:29] interesting pod/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-hj9f6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 03 01:36:44.104917 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:36:44.104749 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-hj9f6" podUID="6b6b29be-71cc-442f-b430-0cc2344d400d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 03 01:36:45.104501 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:36:45.104455 5437 patch_prober.go:29] interesting pod/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-hj9f6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 03 01:36:45.104906 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:36:45.104521 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-hj9f6" podUID="6b6b29be-71cc-442f-b430-0cc2344d400d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 03 01:36:46.103999 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:36:46.103962 5437 patch_prober.go:29] interesting pod/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-hj9f6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 03 01:36:46.104183 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:36:46.104029 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-hj9f6" podUID="6b6b29be-71cc-442f-b430-0cc2344d400d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 03 01:36:47.109068 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:36:47.108996 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-hj9f6" Nov 03 01:36:47.109471 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:36:47.109373 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-hj9f6" Nov 03 01:36:47.110064 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:36:47.110047 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-hj9f6" Nov 03 01:36:47.779719 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:36:47.779669 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:36:52.345986 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:36:52.345951 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-e1793d/e2e-conflict-b-9de9e4b3-kserve-f9bd454-sq6km"] Nov 03 01:36:52.350121 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:36:52.350100 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e1793d/e2e-conflict-b-9de9e4b3-kserve-f9bd454-sq6km" Nov 03 01:36:52.404193 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:36:52.404155 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-e1793d/e2e-conflict-b-9de9e4b3-kserve-f9bd454-sq6km"] Nov 03 01:36:52.453426 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:36:52.453398 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-e1793d\"/\"e2e-conflict-b-9de9e4b3-kserve-self-signed-certs\"" Nov 03 01:36:52.505024 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:36:52.504987 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:36:52.539260 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:36:52.539225 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5f5j\" (UniqueName: \"kubernetes.io/projected/ad9db371-c30e-488a-8182-0a61dcf48926-kube-api-access-b5f5j\") pod \"e2e-conflict-b-9de9e4b3-kserve-f9bd454-sq6km\" (UID: \"ad9db371-c30e-488a-8182-0a61dcf48926\") " pod="e2e-models-e2e-worker-w1-e1793d/e2e-conflict-b-9de9e4b3-kserve-f9bd454-sq6km" Nov 03 01:36:52.539450 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:36:52.539294 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad9db371-c30e-488a-8182-0a61dcf48926-tls-certs\") pod \"e2e-conflict-b-9de9e4b3-kserve-f9bd454-sq6km\" (UID: \"ad9db371-c30e-488a-8182-0a61dcf48926\") " pod="e2e-models-e2e-worker-w1-e1793d/e2e-conflict-b-9de9e4b3-kserve-f9bd454-sq6km" Nov 03 01:36:52.539450 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:36:52.539414 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad9db371-c30e-488a-8182-0a61dcf48926-model-cache\") pod \"e2e-conflict-b-9de9e4b3-kserve-f9bd454-sq6km\" (UID: \"ad9db371-c30e-488a-8182-0a61dcf48926\") " pod="e2e-models-e2e-worker-w1-e1793d/e2e-conflict-b-9de9e4b3-kserve-f9bd454-sq6km" Nov 03 01:36:52.539549 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:36:52.539507 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad9db371-c30e-488a-8182-0a61dcf48926-dshm\") pod \"e2e-conflict-b-9de9e4b3-kserve-f9bd454-sq6km\" (UID: \"ad9db371-c30e-488a-8182-0a61dcf48926\") " pod="e2e-models-e2e-worker-w1-e1793d/e2e-conflict-b-9de9e4b3-kserve-f9bd454-sq6km" Nov 03 01:36:52.539600 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:36:52.539588 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad9db371-c30e-488a-8182-0a61dcf48926-home\") pod \"e2e-conflict-b-9de9e4b3-kserve-f9bd454-sq6km\" (UID: \"ad9db371-c30e-488a-8182-0a61dcf48926\") " pod="e2e-models-e2e-worker-w1-e1793d/e2e-conflict-b-9de9e4b3-kserve-f9bd454-sq6km" Nov 03 01:36:52.539647 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:36:52.539616 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad9db371-c30e-488a-8182-0a61dcf48926-tmp-dir\") pod \"e2e-conflict-b-9de9e4b3-kserve-f9bd454-sq6km\" (UID: \"ad9db371-c30e-488a-8182-0a61dcf48926\") " pod="e2e-models-e2e-worker-w1-e1793d/e2e-conflict-b-9de9e4b3-kserve-f9bd454-sq6km" Nov 03 01:36:52.641293 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:36:52.641204 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad9db371-c30e-488a-8182-0a61dcf48926-dshm\") pod \"e2e-conflict-b-9de9e4b3-kserve-f9bd454-sq6km\" (UID: \"ad9db371-c30e-488a-8182-0a61dcf48926\") " pod="e2e-models-e2e-worker-w1-e1793d/e2e-conflict-b-9de9e4b3-kserve-f9bd454-sq6km" Nov 03 01:36:52.641479 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:36:52.641305 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad9db371-c30e-488a-8182-0a61dcf48926-home\") pod \"e2e-conflict-b-9de9e4b3-kserve-f9bd454-sq6km\" (UID: \"ad9db371-c30e-488a-8182-0a61dcf48926\") " pod="e2e-models-e2e-worker-w1-e1793d/e2e-conflict-b-9de9e4b3-kserve-f9bd454-sq6km" Nov 03 01:36:52.641479 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:36:52.641333 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad9db371-c30e-488a-8182-0a61dcf48926-tmp-dir\") pod \"e2e-conflict-b-9de9e4b3-kserve-f9bd454-sq6km\" (UID: \"ad9db371-c30e-488a-8182-0a61dcf48926\") " pod="e2e-models-e2e-worker-w1-e1793d/e2e-conflict-b-9de9e4b3-kserve-f9bd454-sq6km" Nov 03 01:36:52.641479 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:36:52.641423 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5f5j\" (UniqueName: \"kubernetes.io/projected/ad9db371-c30e-488a-8182-0a61dcf48926-kube-api-access-b5f5j\") pod \"e2e-conflict-b-9de9e4b3-kserve-f9bd454-sq6km\" (UID: \"ad9db371-c30e-488a-8182-0a61dcf48926\") " pod="e2e-models-e2e-worker-w1-e1793d/e2e-conflict-b-9de9e4b3-kserve-f9bd454-sq6km" Nov 03 01:36:52.641479 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:36:52.641462 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad9db371-c30e-488a-8182-0a61dcf48926-tls-certs\") pod \"e2e-conflict-b-9de9e4b3-kserve-f9bd454-sq6km\" (UID: \"ad9db371-c30e-488a-8182-0a61dcf48926\") " pod="e2e-models-e2e-worker-w1-e1793d/e2e-conflict-b-9de9e4b3-kserve-f9bd454-sq6km" Nov 03 01:36:52.641686 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:36:52.641484 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad9db371-c30e-488a-8182-0a61dcf48926-model-cache\") pod \"e2e-conflict-b-9de9e4b3-kserve-f9bd454-sq6km\" (UID: \"ad9db371-c30e-488a-8182-0a61dcf48926\") " pod="e2e-models-e2e-worker-w1-e1793d/e2e-conflict-b-9de9e4b3-kserve-f9bd454-sq6km" Nov 03 01:36:52.641805 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:36:52.641781 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad9db371-c30e-488a-8182-0a61dcf48926-home\") pod \"e2e-conflict-b-9de9e4b3-kserve-f9bd454-sq6km\" (UID: \"ad9db371-c30e-488a-8182-0a61dcf48926\") " pod="e2e-models-e2e-worker-w1-e1793d/e2e-conflict-b-9de9e4b3-kserve-f9bd454-sq6km" Nov 03 01:36:52.641805 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:36:52.641794 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad9db371-c30e-488a-8182-0a61dcf48926-tmp-dir\") pod \"e2e-conflict-b-9de9e4b3-kserve-f9bd454-sq6km\" (UID: \"ad9db371-c30e-488a-8182-0a61dcf48926\") " pod="e2e-models-e2e-worker-w1-e1793d/e2e-conflict-b-9de9e4b3-kserve-f9bd454-sq6km" Nov 03 01:36:52.641915 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:36:52.641861 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad9db371-c30e-488a-8182-0a61dcf48926-model-cache\") pod \"e2e-conflict-b-9de9e4b3-kserve-f9bd454-sq6km\" (UID: \"ad9db371-c30e-488a-8182-0a61dcf48926\") " pod="e2e-models-e2e-worker-w1-e1793d/e2e-conflict-b-9de9e4b3-kserve-f9bd454-sq6km" Nov 03 01:36:52.643516 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:36:52.643493 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad9db371-c30e-488a-8182-0a61dcf48926-dshm\") pod \"e2e-conflict-b-9de9e4b3-kserve-f9bd454-sq6km\" (UID: \"ad9db371-c30e-488a-8182-0a61dcf48926\") " pod="e2e-models-e2e-worker-w1-e1793d/e2e-conflict-b-9de9e4b3-kserve-f9bd454-sq6km" Nov 03 01:36:52.643804 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:36:52.643784 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad9db371-c30e-488a-8182-0a61dcf48926-tls-certs\") pod \"e2e-conflict-b-9de9e4b3-kserve-f9bd454-sq6km\" (UID: \"ad9db371-c30e-488a-8182-0a61dcf48926\") " pod="e2e-models-e2e-worker-w1-e1793d/e2e-conflict-b-9de9e4b3-kserve-f9bd454-sq6km" Nov 03 01:36:52.702962 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:36:52.702924 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:36:52.705003 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:36:52.704976 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5f5j\" (UniqueName: \"kubernetes.io/projected/ad9db371-c30e-488a-8182-0a61dcf48926-kube-api-access-b5f5j\") pod \"e2e-conflict-b-9de9e4b3-kserve-f9bd454-sq6km\" (UID: \"ad9db371-c30e-488a-8182-0a61dcf48926\") " pod="e2e-models-e2e-worker-w1-e1793d/e2e-conflict-b-9de9e4b3-kserve-f9bd454-sq6km" Nov 03 01:36:52.963331 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:36:52.963251 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e1793d/e2e-conflict-b-9de9e4b3-kserve-f9bd454-sq6km" Nov 03 01:36:53.336433 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:36:53.336400 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podad9db371_c30e_488a_8182_0a61dcf48926.slice/crio-ef7db96398cbcc2335142c8fdfa5ee97389f9f3fae52ada1efb0e524b0366e66 WatchSource:0}: Error finding container ef7db96398cbcc2335142c8fdfa5ee97389f9f3fae52ada1efb0e524b0366e66: Status 404 returned error can't find the container with id ef7db96398cbcc2335142c8fdfa5ee97389f9f3fae52ada1efb0e524b0366e66 Nov 03 01:36:53.339156 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:36:53.339132 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-e1793d/e2e-conflict-b-9de9e4b3-kserve-f9bd454-sq6km"] Nov 03 01:36:53.764587 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:36:53.764490 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e1793d/e2e-conflict-b-9de9e4b3-kserve-f9bd454-sq6km" event={"ID":"ad9db371-c30e-488a-8182-0a61dcf48926","Type":"ContainerStarted","Data":"819fb3cd325270a2afd3d3a6a7c82ab0f33bdf027538899e2c1b4a9930971843"} Nov 03 01:36:53.764587 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:36:53.764536 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e1793d/e2e-conflict-b-9de9e4b3-kserve-f9bd454-sq6km" event={"ID":"ad9db371-c30e-488a-8182-0a61dcf48926","Type":"ContainerStarted","Data":"ef7db96398cbcc2335142c8fdfa5ee97389f9f3fae52ada1efb0e524b0366e66"} Nov 03 01:36:53.885328 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:36:53.885277 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-e1793d/e2e-conflict-b-9de9e4b3-kserve-f9bd454-sq6km" podStartSLOduration=1.885263972 podStartE2EDuration="1.885263972s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:36:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:36:53.885003049 +0000 UTC m=+1095.914485356" watchObservedRunningTime="2026-11-03 01:36:53.885263972 +0000 UTC m=+1095.914746266" Nov 03 01:36:53.963425 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:36:53.963380 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e1793d/e2e-conflict-b-9de9e4b3-kserve-f9bd454-sq6km" Nov 03 01:36:57.393150 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:36:57.393110 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:37:02.179863 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:37:02.179823 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:37:02.964253 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:37:02.964213 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-e1793d/e2e-conflict-b-9de9e4b3-kserve-f9bd454-sq6km" Nov 03 01:37:02.986662 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:37:02.986625 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-e1793d/e2e-conflict-b-9de9e4b3-kserve-f9bd454-sq6km" Nov 03 01:37:03.832261 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:37:03.832225 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e1793d/e2e-conflict-b-9de9e4b3-kserve-f9bd454-sq6km" Nov 03 01:37:06.990635 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:37:06.990588 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:37:11.011034 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:37:11.010997 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:37:11.787977 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:37:11.787928 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:37:14.036956 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:37:14.036914 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 01:37:16.586817 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:37:16.586778 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:37:21.389158 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:37:21.389112 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:37:26.192329 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:37:26.192287 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:37:30.985339 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:37:30.985289 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:37:35.782136 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:37:35.782089 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:37:40.586533 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:37:40.586468 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:37:42.898437 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:37:42.898401 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:37:44.035280 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:37:44.035235 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 01:37:45.382330 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:37:45.382238 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:37:46.133443 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:37:46.133409 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-m26wn"] Nov 03 01:37:46.137769 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:37:46.137748 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-m26wn" Nov 03 01:37:46.171241 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:37:46.171203 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71058df3-a083-440b-b68a-5dd686509a66-istio-token\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-m26wn\" (UID: \"71058df3-a083-440b-b68a-5dd686509a66\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-m26wn" Nov 03 01:37:46.171434 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:37:46.171288 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71058df3-a083-440b-b68a-5dd686509a66-istio-envoy\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-m26wn\" (UID: \"71058df3-a083-440b-b68a-5dd686509a66\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-m26wn" Nov 03 01:37:46.171508 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:37:46.171488 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71058df3-a083-440b-b68a-5dd686509a66-workload-socket\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-m26wn\" (UID: \"71058df3-a083-440b-b68a-5dd686509a66\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-m26wn" Nov 03 01:37:46.171559 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:37:46.171525 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71058df3-a083-440b-b68a-5dd686509a66-istio-data\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-m26wn\" (UID: \"71058df3-a083-440b-b68a-5dd686509a66\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-m26wn" Nov 03 01:37:46.171597 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:37:46.171558 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71058df3-a083-440b-b68a-5dd686509a66-istiod-ca-cert\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-m26wn\" (UID: \"71058df3-a083-440b-b68a-5dd686509a66\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-m26wn" Nov 03 01:37:46.171654 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:37:46.171597 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71058df3-a083-440b-b68a-5dd686509a66-credential-socket\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-m26wn\" (UID: \"71058df3-a083-440b-b68a-5dd686509a66\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-m26wn" Nov 03 01:37:46.171722 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:37:46.171691 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71058df3-a083-440b-b68a-5dd686509a66-istio-podinfo\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-m26wn\" (UID: \"71058df3-a083-440b-b68a-5dd686509a66\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-m26wn" Nov 03 01:37:46.171809 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:37:46.171794 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2h22m\" (UniqueName: \"kubernetes.io/projected/71058df3-a083-440b-b68a-5dd686509a66-kube-api-access-2h22m\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-m26wn\" (UID: \"71058df3-a083-440b-b68a-5dd686509a66\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-m26wn" Nov 03 01:37:46.171867 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:37:46.171843 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71058df3-a083-440b-b68a-5dd686509a66-workload-certs\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-m26wn\" (UID: \"71058df3-a083-440b-b68a-5dd686509a66\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-m26wn" Nov 03 01:37:46.234726 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:37:46.234687 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-m26wn"] Nov 03 01:37:46.272732 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:37:46.272696 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71058df3-a083-440b-b68a-5dd686509a66-istio-token\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-m26wn\" (UID: \"71058df3-a083-440b-b68a-5dd686509a66\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-m26wn" Nov 03 01:37:46.272921 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:37:46.272754 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71058df3-a083-440b-b68a-5dd686509a66-istio-envoy\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-m26wn\" (UID: \"71058df3-a083-440b-b68a-5dd686509a66\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-m26wn" Nov 03 01:37:46.272921 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:37:46.272797 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71058df3-a083-440b-b68a-5dd686509a66-workload-socket\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-m26wn\" (UID: \"71058df3-a083-440b-b68a-5dd686509a66\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-m26wn" Nov 03 01:37:46.272921 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:37:46.272822 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71058df3-a083-440b-b68a-5dd686509a66-istio-data\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-m26wn\" (UID: \"71058df3-a083-440b-b68a-5dd686509a66\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-m26wn" Nov 03 01:37:46.272921 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:37:46.272843 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71058df3-a083-440b-b68a-5dd686509a66-istiod-ca-cert\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-m26wn\" (UID: \"71058df3-a083-440b-b68a-5dd686509a66\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-m26wn" Nov 03 01:37:46.272921 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:37:46.272862 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71058df3-a083-440b-b68a-5dd686509a66-credential-socket\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-m26wn\" (UID: \"71058df3-a083-440b-b68a-5dd686509a66\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-m26wn" Nov 03 01:37:46.273176 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:37:46.272932 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71058df3-a083-440b-b68a-5dd686509a66-istio-podinfo\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-m26wn\" (UID: \"71058df3-a083-440b-b68a-5dd686509a66\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-m26wn" Nov 03 01:37:46.273176 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:37:46.273045 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2h22m\" (UniqueName: \"kubernetes.io/projected/71058df3-a083-440b-b68a-5dd686509a66-kube-api-access-2h22m\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-m26wn\" (UID: \"71058df3-a083-440b-b68a-5dd686509a66\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-m26wn" Nov 03 01:37:46.273176 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:37:46.273095 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71058df3-a083-440b-b68a-5dd686509a66-workload-certs\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-m26wn\" (UID: \"71058df3-a083-440b-b68a-5dd686509a66\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-m26wn" Nov 03 01:37:46.273340 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:37:46.273223 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71058df3-a083-440b-b68a-5dd686509a66-workload-socket\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-m26wn\" (UID: \"71058df3-a083-440b-b68a-5dd686509a66\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-m26wn" Nov 03 01:37:46.273416 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:37:46.273342 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71058df3-a083-440b-b68a-5dd686509a66-istio-data\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-m26wn\" (UID: \"71058df3-a083-440b-b68a-5dd686509a66\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-m26wn" Nov 03 01:37:46.273416 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:37:46.273400 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71058df3-a083-440b-b68a-5dd686509a66-credential-socket\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-m26wn\" (UID: \"71058df3-a083-440b-b68a-5dd686509a66\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-m26wn" Nov 03 01:37:46.273536 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:37:46.273517 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71058df3-a083-440b-b68a-5dd686509a66-workload-certs\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-m26wn\" (UID: \"71058df3-a083-440b-b68a-5dd686509a66\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-m26wn" Nov 03 01:37:46.273702 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:37:46.273678 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71058df3-a083-440b-b68a-5dd686509a66-istiod-ca-cert\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-m26wn\" (UID: \"71058df3-a083-440b-b68a-5dd686509a66\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-m26wn" Nov 03 01:37:46.275141 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:37:46.275121 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71058df3-a083-440b-b68a-5dd686509a66-istio-envoy\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-m26wn\" (UID: \"71058df3-a083-440b-b68a-5dd686509a66\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-m26wn" Nov 03 01:37:46.275477 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:37:46.275458 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71058df3-a083-440b-b68a-5dd686509a66-istio-podinfo\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-m26wn\" (UID: \"71058df3-a083-440b-b68a-5dd686509a66\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-m26wn" Nov 03 01:37:46.335875 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:37:46.335839 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71058df3-a083-440b-b68a-5dd686509a66-istio-token\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-m26wn\" (UID: \"71058df3-a083-440b-b68a-5dd686509a66\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-m26wn" Nov 03 01:37:46.335875 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:37:46.335868 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2h22m\" (UniqueName: \"kubernetes.io/projected/71058df3-a083-440b-b68a-5dd686509a66-kube-api-access-2h22m\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-m26wn\" (UID: \"71058df3-a083-440b-b68a-5dd686509a66\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-m26wn" Nov 03 01:37:46.451676 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:37:46.451589 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-m26wn" Nov 03 01:37:46.872679 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:37:46.872644 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71058df3_a083_440b_b68a_5dd686509a66.slice/crio-c0330276fc2a0b5554fb57658cb6006f4b2f3c1d1676c16f2634628d707171bf WatchSource:0}: Error finding container c0330276fc2a0b5554fb57658cb6006f4b2f3c1d1676c16f2634628d707171bf: Status 404 returned error can't find the container with id c0330276fc2a0b5554fb57658cb6006f4b2f3c1d1676c16f2634628d707171bf Nov 03 01:37:46.875099 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:37:46.875059 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 01:37:46.875214 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:37:46.875146 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 01:37:46.875214 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:37:46.875192 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 01:37:46.875328 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:37:46.875243 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-m26wn"] Nov 03 01:37:47.082239 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:37:47.082203 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-m26wn" event={"ID":"71058df3-a083-440b-b68a-5dd686509a66","Type":"ContainerStarted","Data":"21cf4f2e51142980b59d8e8f7f18a274a46eeb54c6305d7703ae7e12f38862dd"} Nov 03 01:37:47.082638 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:37:47.082258 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-m26wn" event={"ID":"71058df3-a083-440b-b68a-5dd686509a66","Type":"ContainerStarted","Data":"c0330276fc2a0b5554fb57658cb6006f4b2f3c1d1676c16f2634628d707171bf"} Nov 03 01:37:47.108516 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:37:47.108478 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:37:47.208236 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:37:47.208134 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-m26wn" podStartSLOduration=1.208117887 podStartE2EDuration="1.208117887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:37:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:37:47.206085783 +0000 UTC m=+1149.235568078" watchObservedRunningTime="2026-11-03 01:37:47.208117887 +0000 UTC m=+1149.237600182" Nov 03 01:37:47.309897 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:37:47.309867 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:37:47.452325 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:37:47.452289 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-m26wn" Nov 03 01:37:47.454152 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:37:47.454122 5437 patch_prober.go:29] interesting pod/e2e-worker-w2-9da431-openshift-default-65c799fd77-m26wn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 03 01:37:47.454297 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:37:47.454181 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-m26wn" podUID="71058df3-a083-440b-b68a-5dd686509a66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 03 01:37:47.518019 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:37:47.517934 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:37:47.880823 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:37:47.880786 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:37:48.452736 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:37:48.452698 5437 patch_prober.go:29] interesting pod/e2e-worker-w2-9da431-openshift-default-65c799fd77-m26wn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 03 01:37:48.453189 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:37:48.452775 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-m26wn" podUID="71058df3-a083-440b-b68a-5dd686509a66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 03 01:37:49.452062 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:37:49.452013 5437 patch_prober.go:29] interesting pod/e2e-worker-w2-9da431-openshift-default-65c799fd77-m26wn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 03 01:37:49.452266 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:37:49.452086 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-m26wn" podUID="71058df3-a083-440b-b68a-5dd686509a66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 03 01:37:50.215067 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:37:50.215033 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:37:50.452360 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:37:50.452321 5437 patch_prober.go:29] interesting pod/e2e-worker-w2-9da431-openshift-default-65c799fd77-m26wn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 03 01:37:50.452632 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:37:50.452602 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-m26wn" podUID="71058df3-a083-440b-b68a-5dd686509a66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 03 01:37:51.452144 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:37:51.452094 5437 patch_prober.go:29] interesting pod/e2e-worker-w2-9da431-openshift-default-65c799fd77-m26wn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 03 01:37:51.452588 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:37:51.452159 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-m26wn" podUID="71058df3-a083-440b-b68a-5dd686509a66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 03 01:37:52.452432 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:37:52.452393 5437 patch_prober.go:29] interesting pod/e2e-worker-w2-9da431-openshift-default-65c799fd77-m26wn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 03 01:37:52.452860 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:37:52.452447 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-m26wn" podUID="71058df3-a083-440b-b68a-5dd686509a66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 03 01:37:53.452704 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:37:53.452660 5437 patch_prober.go:29] interesting pod/e2e-worker-w2-9da431-openshift-default-65c799fd77-m26wn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 03 01:37:53.453231 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:37:53.452741 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-m26wn" podUID="71058df3-a083-440b-b68a-5dd686509a66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 03 01:37:54.456880 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:37:54.456841 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-m26wn" Nov 03 01:37:54.457459 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:37:54.457268 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-m26wn" Nov 03 01:37:54.458228 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:37:54.458207 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-m26wn" Nov 03 01:37:55.093659 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:37:55.093610 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:37:57.346461 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:37:57.346400 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:37:57.595918 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:37:57.595875 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:37:59.170849 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:37:59.170802 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:37:59.889646 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:37:59.889609 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:38:04.670982 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:04.670934 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:38:09.470783 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:09.470738 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:38:11.720074 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:11.720035 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-e1793d/e2e-conflict-b-9de9e4b3-kserve-f9bd454-sq6km"] Nov 03 01:38:11.841134 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:11.841057 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:38:14.040171 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:38:14.040136 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 01:38:14.275417 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:14.275353 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:38:19.017612 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:19.017483 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:38:23.797268 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:23.797227 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:38:26.740550 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:26.740512 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-e1793d/e2e-conflict-b-9de9e4b3-kserve-f9bd454-sq6km" podUID="ad9db371-c30e-488a-8182-0a61dcf48926" containerName="main" containerID="cri-o://819fb3cd325270a2afd3d3a6a7c82ab0f33bdf027538899e2c1b4a9930971843" gracePeriod=45 Nov 03 01:38:26.820787 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:26.820749 5437 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e1793d/e2e-conflict-b-9de9e4b3-kserve-f9bd454-sq6km" podUID="ad9db371-c30e-488a-8182-0a61dcf48926" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": dial tcp 10.134.0.73:8000: connect: connection refused" Nov 03 01:38:27.017976 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:27.017950 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e1793d/e2e-conflict-b-9de9e4b3-kserve-f9bd454-sq6km" Nov 03 01:38:27.090468 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:27.090433 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad9db371-c30e-488a-8182-0a61dcf48926-kube-api-access-b5f5j\" (UniqueName: \"kubernetes.io/projected/ad9db371-c30e-488a-8182-0a61dcf48926-kube-api-access-b5f5j\") pod \"ad9db371-c30e-488a-8182-0a61dcf48926\" (UID: \"ad9db371-c30e-488a-8182-0a61dcf48926\") " Nov 03 01:38:27.090689 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:27.090494 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad9db371-c30e-488a-8182-0a61dcf48926-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad9db371-c30e-488a-8182-0a61dcf48926-model-cache\") pod \"ad9db371-c30e-488a-8182-0a61dcf48926\" (UID: \"ad9db371-c30e-488a-8182-0a61dcf48926\") " Nov 03 01:38:27.090689 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:27.090569 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad9db371-c30e-488a-8182-0a61dcf48926-home\" (UniqueName: \"kubernetes.io/empty-dir/ad9db371-c30e-488a-8182-0a61dcf48926-home\") pod \"ad9db371-c30e-488a-8182-0a61dcf48926\" (UID: \"ad9db371-c30e-488a-8182-0a61dcf48926\") " Nov 03 01:38:27.090689 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:27.090634 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad9db371-c30e-488a-8182-0a61dcf48926-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad9db371-c30e-488a-8182-0a61dcf48926-tmp-dir\") pod \"ad9db371-c30e-488a-8182-0a61dcf48926\" (UID: \"ad9db371-c30e-488a-8182-0a61dcf48926\") " Nov 03 01:38:27.090689 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:27.090681 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad9db371-c30e-488a-8182-0a61dcf48926-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad9db371-c30e-488a-8182-0a61dcf48926-dshm\") pod \"ad9db371-c30e-488a-8182-0a61dcf48926\" (UID: \"ad9db371-c30e-488a-8182-0a61dcf48926\") " Nov 03 01:38:27.090902 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:27.090757 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad9db371-c30e-488a-8182-0a61dcf48926-tls-certs\" (UniqueName: \"kubernetes.io/secret/ad9db371-c30e-488a-8182-0a61dcf48926-tls-certs\") pod \"ad9db371-c30e-488a-8182-0a61dcf48926\" (UID: \"ad9db371-c30e-488a-8182-0a61dcf48926\") " Nov 03 01:38:27.090902 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:27.090777 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad9db371-c30e-488a-8182-0a61dcf48926-model-cache" pod "ad9db371-c30e-488a-8182-0a61dcf48926" (UID: "ad9db371-c30e-488a-8182-0a61dcf48926"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:38:27.090902 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:27.090878 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad9db371-c30e-488a-8182-0a61dcf48926-home" pod "ad9db371-c30e-488a-8182-0a61dcf48926" (UID: "ad9db371-c30e-488a-8182-0a61dcf48926"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:38:27.091056 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:27.090923 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad9db371-c30e-488a-8182-0a61dcf48926-tmp-dir" pod "ad9db371-c30e-488a-8182-0a61dcf48926" (UID: "ad9db371-c30e-488a-8182-0a61dcf48926"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:38:27.091267 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:27.091243 5437 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad9db371-c30e-488a-8182-0a61dcf48926-tmp-dir\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:38:27.091418 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:27.091271 5437 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad9db371-c30e-488a-8182-0a61dcf48926-model-cache\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:38:27.091418 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:27.091286 5437 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad9db371-c30e-488a-8182-0a61dcf48926-home\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:38:27.093033 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:27.093003 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad9db371-c30e-488a-8182-0a61dcf48926-dshm" pod "ad9db371-c30e-488a-8182-0a61dcf48926" (UID: "ad9db371-c30e-488a-8182-0a61dcf48926"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:38:27.093138 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:27.093031 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad9db371-c30e-488a-8182-0a61dcf48926-kube-api-access-b5f5j" pod "ad9db371-c30e-488a-8182-0a61dcf48926" (UID: "ad9db371-c30e-488a-8182-0a61dcf48926"). InnerVolumeSpecName "kube-api-access-b5f5j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:38:27.093138 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:27.093033 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad9db371-c30e-488a-8182-0a61dcf48926-tls-certs" pod "ad9db371-c30e-488a-8182-0a61dcf48926" (UID: "ad9db371-c30e-488a-8182-0a61dcf48926"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:38:27.193031 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:27.192989 5437 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad9db371-c30e-488a-8182-0a61dcf48926-dshm\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:38:27.193193 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:27.193102 5437 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad9db371-c30e-488a-8182-0a61dcf48926-tls-certs\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:38:27.193193 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:27.193120 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5f5j\" (UniqueName: \"kubernetes.io/projected/ad9db371-c30e-488a-8182-0a61dcf48926-kube-api-access-b5f5j\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:38:27.346639 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:27.346598 5437 generic.go:356] "Generic (PLEG): container finished" podID="ad9db371-c30e-488a-8182-0a61dcf48926" containerID="819fb3cd325270a2afd3d3a6a7c82ab0f33bdf027538899e2c1b4a9930971843" exitCode=0 Nov 03 01:38:27.346807 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:27.346685 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e1793d/e2e-conflict-b-9de9e4b3-kserve-f9bd454-sq6km" event={"ID":"ad9db371-c30e-488a-8182-0a61dcf48926","Type":"ContainerDied","Data":"819fb3cd325270a2afd3d3a6a7c82ab0f33bdf027538899e2c1b4a9930971843"} Nov 03 01:38:27.346807 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:27.346734 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e1793d/e2e-conflict-b-9de9e4b3-kserve-f9bd454-sq6km" event={"ID":"ad9db371-c30e-488a-8182-0a61dcf48926","Type":"ContainerDied","Data":"ef7db96398cbcc2335142c8fdfa5ee97389f9f3fae52ada1efb0e524b0366e66"} Nov 03 01:38:27.346807 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:27.346753 5437 scope.go:122] "RemoveContainer" containerID="819fb3cd325270a2afd3d3a6a7c82ab0f33bdf027538899e2c1b4a9930971843" Nov 03 01:38:27.346807 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:27.346763 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e1793d/e2e-conflict-b-9de9e4b3-kserve-f9bd454-sq6km" Nov 03 01:38:27.357222 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:27.357204 5437 scope.go:122] "RemoveContainer" containerID="819fb3cd325270a2afd3d3a6a7c82ab0f33bdf027538899e2c1b4a9930971843" Nov 03 01:38:27.357576 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:38:27.357548 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"819fb3cd325270a2afd3d3a6a7c82ab0f33bdf027538899e2c1b4a9930971843\": container with ID starting with 819fb3cd325270a2afd3d3a6a7c82ab0f33bdf027538899e2c1b4a9930971843 not found: ID does not exist" containerID="819fb3cd325270a2afd3d3a6a7c82ab0f33bdf027538899e2c1b4a9930971843" Nov 03 01:38:27.357647 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:27.357589 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"819fb3cd325270a2afd3d3a6a7c82ab0f33bdf027538899e2c1b4a9930971843"} err="failed to get container status \"819fb3cd325270a2afd3d3a6a7c82ab0f33bdf027538899e2c1b4a9930971843\": rpc error: code = NotFound desc = could not find container \"819fb3cd325270a2afd3d3a6a7c82ab0f33bdf027538899e2c1b4a9930971843\": container with ID starting with 819fb3cd325270a2afd3d3a6a7c82ab0f33bdf027538899e2c1b4a9930971843 not found: ID does not exist" Nov 03 01:38:27.517971 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:27.517940 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-e1793d/e2e-conflict-b-9de9e4b3-kserve-f9bd454-sq6km"] Nov 03 01:38:27.522089 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:27.522061 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-e1793d/e2e-conflict-b-9de9e4b3-kserve-f9bd454-sq6km"] Nov 03 01:38:28.088584 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:28.088548 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad9db371-c30e-488a-8182-0a61dcf48926" path="/var/lib/kubelet/pods/ad9db371-c30e-488a-8182-0a61dcf48926/volumes" Nov 03 01:38:28.668105 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:28.668072 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:38:30.542418 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:30.477148 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-e1793d/e2e-conflict-a-9de9e4b3-kserve-79884b557b-bp2sj"] Nov 03 01:38:30.542418 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:30.503038 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:38:33.581093 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:33.581051 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:38:37.944444 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:37.944405 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-zjbzg"] Nov 03 01:38:37.945058 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:37.945043 5437 cpu_manager.go:414] "containerMap: removing container" podUID="ad9db371-c30e-488a-8182-0a61dcf48926" containerName="main" Nov 03 01:38:37.945107 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:37.945060 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad9db371-c30e-488a-8182-0a61dcf48926" containerName="main" Nov 03 01:38:37.945169 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:37.945159 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad9db371-c30e-488a-8182-0a61dcf48926" containerName="main" Nov 03 01:38:37.949714 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:37.949697 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-zjbzg" Nov 03 01:38:38.004281 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:38.004249 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-0bf34a-openshift-default-dockercfg-s9n2w\"" Nov 03 01:38:38.029974 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:38.029934 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88142666-5b4b-40db-815e-73c0d8db0993-credential-socket\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-zjbzg\" (UID: \"88142666-5b4b-40db-815e-73c0d8db0993\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-zjbzg" Nov 03 01:38:38.030118 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:38.029989 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88142666-5b4b-40db-815e-73c0d8db0993-workload-socket\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-zjbzg\" (UID: \"88142666-5b4b-40db-815e-73c0d8db0993\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-zjbzg" Nov 03 01:38:38.030176 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:38.030112 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88142666-5b4b-40db-815e-73c0d8db0993-istio-data\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-zjbzg\" (UID: \"88142666-5b4b-40db-815e-73c0d8db0993\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-zjbzg" Nov 03 01:38:38.030176 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:38.030142 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88142666-5b4b-40db-815e-73c0d8db0993-istiod-ca-cert\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-zjbzg\" (UID: \"88142666-5b4b-40db-815e-73c0d8db0993\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-zjbzg" Nov 03 01:38:38.030243 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:38.030235 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j66qj\" (UniqueName: \"kubernetes.io/projected/88142666-5b4b-40db-815e-73c0d8db0993-kube-api-access-j66qj\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-zjbzg\" (UID: \"88142666-5b4b-40db-815e-73c0d8db0993\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-zjbzg" Nov 03 01:38:38.030292 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:38.030279 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88142666-5b4b-40db-815e-73c0d8db0993-istio-podinfo\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-zjbzg\" (UID: \"88142666-5b4b-40db-815e-73c0d8db0993\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-zjbzg" Nov 03 01:38:38.030355 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:38.030343 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88142666-5b4b-40db-815e-73c0d8db0993-istio-envoy\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-zjbzg\" (UID: \"88142666-5b4b-40db-815e-73c0d8db0993\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-zjbzg" Nov 03 01:38:38.030453 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:38.030440 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88142666-5b4b-40db-815e-73c0d8db0993-istio-token\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-zjbzg\" (UID: \"88142666-5b4b-40db-815e-73c0d8db0993\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-zjbzg" Nov 03 01:38:38.030537 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:38.030515 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88142666-5b4b-40db-815e-73c0d8db0993-workload-certs\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-zjbzg\" (UID: \"88142666-5b4b-40db-815e-73c0d8db0993\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-zjbzg" Nov 03 01:38:38.059741 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:38.059699 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-zjbzg"] Nov 03 01:38:38.131587 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:38.131547 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j66qj\" (UniqueName: \"kubernetes.io/projected/88142666-5b4b-40db-815e-73c0d8db0993-kube-api-access-j66qj\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-zjbzg\" (UID: \"88142666-5b4b-40db-815e-73c0d8db0993\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-zjbzg" Nov 03 01:38:38.131587 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:38.131587 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88142666-5b4b-40db-815e-73c0d8db0993-istio-podinfo\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-zjbzg\" (UID: \"88142666-5b4b-40db-815e-73c0d8db0993\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-zjbzg" Nov 03 01:38:38.131854 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:38.131626 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88142666-5b4b-40db-815e-73c0d8db0993-istio-envoy\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-zjbzg\" (UID: \"88142666-5b4b-40db-815e-73c0d8db0993\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-zjbzg" Nov 03 01:38:38.131854 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:38.131694 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88142666-5b4b-40db-815e-73c0d8db0993-istio-token\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-zjbzg\" (UID: \"88142666-5b4b-40db-815e-73c0d8db0993\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-zjbzg" Nov 03 01:38:38.131854 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:38.131734 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88142666-5b4b-40db-815e-73c0d8db0993-workload-certs\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-zjbzg\" (UID: \"88142666-5b4b-40db-815e-73c0d8db0993\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-zjbzg" Nov 03 01:38:38.131854 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:38.131807 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88142666-5b4b-40db-815e-73c0d8db0993-credential-socket\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-zjbzg\" (UID: \"88142666-5b4b-40db-815e-73c0d8db0993\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-zjbzg" Nov 03 01:38:38.131854 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:38.131842 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88142666-5b4b-40db-815e-73c0d8db0993-workload-socket\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-zjbzg\" (UID: \"88142666-5b4b-40db-815e-73c0d8db0993\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-zjbzg" Nov 03 01:38:38.132155 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:38.131875 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88142666-5b4b-40db-815e-73c0d8db0993-istio-data\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-zjbzg\" (UID: \"88142666-5b4b-40db-815e-73c0d8db0993\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-zjbzg" Nov 03 01:38:38.132155 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:38.131897 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88142666-5b4b-40db-815e-73c0d8db0993-istiod-ca-cert\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-zjbzg\" (UID: \"88142666-5b4b-40db-815e-73c0d8db0993\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-zjbzg" Nov 03 01:38:38.132260 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:38.132193 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88142666-5b4b-40db-815e-73c0d8db0993-workload-certs\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-zjbzg\" (UID: \"88142666-5b4b-40db-815e-73c0d8db0993\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-zjbzg" Nov 03 01:38:38.132260 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:38.132228 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88142666-5b4b-40db-815e-73c0d8db0993-credential-socket\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-zjbzg\" (UID: \"88142666-5b4b-40db-815e-73c0d8db0993\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-zjbzg" Nov 03 01:38:38.132385 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:38.132314 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88142666-5b4b-40db-815e-73c0d8db0993-workload-socket\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-zjbzg\" (UID: \"88142666-5b4b-40db-815e-73c0d8db0993\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-zjbzg" Nov 03 01:38:38.132385 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:38.132350 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88142666-5b4b-40db-815e-73c0d8db0993-istio-data\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-zjbzg\" (UID: \"88142666-5b4b-40db-815e-73c0d8db0993\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-zjbzg" Nov 03 01:38:38.132644 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:38.132622 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88142666-5b4b-40db-815e-73c0d8db0993-istiod-ca-cert\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-zjbzg\" (UID: \"88142666-5b4b-40db-815e-73c0d8db0993\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-zjbzg" Nov 03 01:38:38.134092 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:38.134069 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88142666-5b4b-40db-815e-73c0d8db0993-istio-envoy\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-zjbzg\" (UID: \"88142666-5b4b-40db-815e-73c0d8db0993\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-zjbzg" Nov 03 01:38:38.134289 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:38.134272 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88142666-5b4b-40db-815e-73c0d8db0993-istio-podinfo\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-zjbzg\" (UID: \"88142666-5b4b-40db-815e-73c0d8db0993\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-zjbzg" Nov 03 01:38:38.195693 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:38.195622 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88142666-5b4b-40db-815e-73c0d8db0993-istio-token\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-zjbzg\" (UID: \"88142666-5b4b-40db-815e-73c0d8db0993\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-zjbzg" Nov 03 01:38:38.195967 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:38.195940 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j66qj\" (UniqueName: \"kubernetes.io/projected/88142666-5b4b-40db-815e-73c0d8db0993-kube-api-access-j66qj\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-zjbzg\" (UID: \"88142666-5b4b-40db-815e-73c0d8db0993\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-zjbzg" Nov 03 01:38:38.317285 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:38.317259 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-0bf34a-openshift-default-dockercfg-s9n2w\"" Nov 03 01:38:38.324417 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:38.324392 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-zjbzg" Nov 03 01:38:38.392154 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:38.391977 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:38:38.755196 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:38:38.755160 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88142666_5b4b_40db_815e_73c0d8db0993.slice/crio-3da511ced694af6a9258aebcf517d401bdc715f4fdde532ebcc3a37be975b4ad WatchSource:0}: Error finding container 3da511ced694af6a9258aebcf517d401bdc715f4fdde532ebcc3a37be975b4ad: Status 404 returned error can't find the container with id 3da511ced694af6a9258aebcf517d401bdc715f4fdde532ebcc3a37be975b4ad Nov 03 01:38:38.758319 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:38.758282 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 01:38:38.758319 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:38.758313 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-zjbzg"] Nov 03 01:38:38.758522 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:38.758380 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 01:38:38.758522 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:38.758428 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 01:38:39.465315 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:39.465279 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-zjbzg" event={"ID":"88142666-5b4b-40db-815e-73c0d8db0993","Type":"ContainerStarted","Data":"44084a92134019fa7c9d68c0f465ad6ff2a72259c571adc9774c53756938ab2e"} Nov 03 01:38:39.465852 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:39.465332 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-zjbzg" event={"ID":"88142666-5b4b-40db-815e-73c0d8db0993","Type":"ContainerStarted","Data":"3da511ced694af6a9258aebcf517d401bdc715f4fdde532ebcc3a37be975b4ad"} Nov 03 01:38:39.650993 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:39.650931 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-zjbzg" podStartSLOduration=2.650915607 podStartE2EDuration="2.650915607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:38:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:38:39.648741819 +0000 UTC m=+1201.678224113" watchObservedRunningTime="2026-11-03 01:38:39.650915607 +0000 UTC m=+1201.680397901" Nov 03 01:38:40.339412 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:40.338773 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-zjbzg" Nov 03 01:38:40.343909 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:40.343871 5437 patch_prober.go:29] interesting pod/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-zjbzg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 03 01:38:40.344118 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:40.343952 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-zjbzg" podUID="88142666-5b4b-40db-815e-73c0d8db0993" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 03 01:38:41.335906 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:41.335860 5437 patch_prober.go:29] interesting pod/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-zjbzg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 03 01:38:41.336581 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:41.336542 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-zjbzg" podUID="88142666-5b4b-40db-815e-73c0d8db0993" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 03 01:38:42.343943 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:42.343902 5437 patch_prober.go:29] interesting pod/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-zjbzg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 03 01:38:42.344582 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:42.344546 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-zjbzg" podUID="88142666-5b4b-40db-815e-73c0d8db0993" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 03 01:38:42.409353 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:42.409299 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:38:43.216425 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:43.216351 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:38:43.335393 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:43.334834 5437 patch_prober.go:29] interesting pod/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-zjbzg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 03 01:38:43.335393 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:43.334919 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-zjbzg" podUID="88142666-5b4b-40db-815e-73c0d8db0993" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 03 01:38:44.057694 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:38:44.057654 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 01:38:44.342590 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:44.342491 5437 patch_prober.go:29] interesting pod/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-zjbzg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 03 01:38:44.342590 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:44.342572 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-zjbzg" podUID="88142666-5b4b-40db-815e-73c0d8db0993" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 03 01:38:45.325827 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:45.325781 5437 patch_prober.go:29] interesting pod/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-zjbzg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 03 01:38:45.326259 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:45.325859 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-zjbzg" podUID="88142666-5b4b-40db-815e-73c0d8db0993" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 03 01:38:45.614643 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:45.614538 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-e1793d/e2e-conflict-a-9de9e4b3-kserve-79884b557b-bp2sj" podUID="9a290bd4-9950-4806-bf86-1f02c8b60ea6" containerName="main" containerID="cri-o://e34f717cfe4652199ac2027d0d735950b648d20662872f890f6818171dfb3ac0" gracePeriod=45 Nov 03 01:38:45.895843 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:45.895804 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e1793d/e2e-conflict-a-9de9e4b3-kserve-79884b557b-bp2sj" Nov 03 01:38:46.022484 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:46.022440 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a290bd4-9950-4806-bf86-1f02c8b60ea6-kube-api-access-465d4\" (UniqueName: \"kubernetes.io/projected/9a290bd4-9950-4806-bf86-1f02c8b60ea6-kube-api-access-465d4\") pod \"9a290bd4-9950-4806-bf86-1f02c8b60ea6\" (UID: \"9a290bd4-9950-4806-bf86-1f02c8b60ea6\") " Nov 03 01:38:46.022698 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:46.022560 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a290bd4-9950-4806-bf86-1f02c8b60ea6-home\" (UniqueName: \"kubernetes.io/empty-dir/9a290bd4-9950-4806-bf86-1f02c8b60ea6-home\") pod \"9a290bd4-9950-4806-bf86-1f02c8b60ea6\" (UID: \"9a290bd4-9950-4806-bf86-1f02c8b60ea6\") " Nov 03 01:38:46.022698 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:46.022610 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a290bd4-9950-4806-bf86-1f02c8b60ea6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a290bd4-9950-4806-bf86-1f02c8b60ea6-model-cache\") pod \"9a290bd4-9950-4806-bf86-1f02c8b60ea6\" (UID: \"9a290bd4-9950-4806-bf86-1f02c8b60ea6\") " Nov 03 01:38:46.022698 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:46.022657 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a290bd4-9950-4806-bf86-1f02c8b60ea6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a290bd4-9950-4806-bf86-1f02c8b60ea6-dshm\") pod \"9a290bd4-9950-4806-bf86-1f02c8b60ea6\" (UID: \"9a290bd4-9950-4806-bf86-1f02c8b60ea6\") " Nov 03 01:38:46.022891 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:46.022728 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a290bd4-9950-4806-bf86-1f02c8b60ea6-tls-certs\" (UniqueName: \"kubernetes.io/secret/9a290bd4-9950-4806-bf86-1f02c8b60ea6-tls-certs\") pod \"9a290bd4-9950-4806-bf86-1f02c8b60ea6\" (UID: \"9a290bd4-9950-4806-bf86-1f02c8b60ea6\") " Nov 03 01:38:46.022891 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:46.022812 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a290bd4-9950-4806-bf86-1f02c8b60ea6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a290bd4-9950-4806-bf86-1f02c8b60ea6-tmp-dir\") pod \"9a290bd4-9950-4806-bf86-1f02c8b60ea6\" (UID: \"9a290bd4-9950-4806-bf86-1f02c8b60ea6\") " Nov 03 01:38:46.023412 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:46.023059 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a290bd4-9950-4806-bf86-1f02c8b60ea6-model-cache" pod "9a290bd4-9950-4806-bf86-1f02c8b60ea6" (UID: "9a290bd4-9950-4806-bf86-1f02c8b60ea6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:38:46.023412 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:46.023231 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a290bd4-9950-4806-bf86-1f02c8b60ea6-home" pod "9a290bd4-9950-4806-bf86-1f02c8b60ea6" (UID: "9a290bd4-9950-4806-bf86-1f02c8b60ea6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:38:46.023412 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:46.023262 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a290bd4-9950-4806-bf86-1f02c8b60ea6-tmp-dir" pod "9a290bd4-9950-4806-bf86-1f02c8b60ea6" (UID: "9a290bd4-9950-4806-bf86-1f02c8b60ea6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:38:46.023412 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:46.023286 5437 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a290bd4-9950-4806-bf86-1f02c8b60ea6-model-cache\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:38:46.026005 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:46.025961 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a290bd4-9950-4806-bf86-1f02c8b60ea6-kube-api-access-465d4" pod "9a290bd4-9950-4806-bf86-1f02c8b60ea6" (UID: "9a290bd4-9950-4806-bf86-1f02c8b60ea6"). InnerVolumeSpecName "kube-api-access-465d4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:38:46.026167 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:46.025968 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a290bd4-9950-4806-bf86-1f02c8b60ea6-tls-certs" pod "9a290bd4-9950-4806-bf86-1f02c8b60ea6" (UID: "9a290bd4-9950-4806-bf86-1f02c8b60ea6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:38:46.026343 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:46.026313 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a290bd4-9950-4806-bf86-1f02c8b60ea6-dshm" pod "9a290bd4-9950-4806-bf86-1f02c8b60ea6" (UID: "9a290bd4-9950-4806-bf86-1f02c8b60ea6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:38:46.125109 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:46.125011 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-465d4\" (UniqueName: \"kubernetes.io/projected/9a290bd4-9950-4806-bf86-1f02c8b60ea6-kube-api-access-465d4\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:38:46.125109 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:46.125048 5437 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a290bd4-9950-4806-bf86-1f02c8b60ea6-home\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:38:46.125109 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:46.125061 5437 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a290bd4-9950-4806-bf86-1f02c8b60ea6-dshm\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:38:46.125109 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:46.125069 5437 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a290bd4-9950-4806-bf86-1f02c8b60ea6-tls-certs\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:38:46.125109 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:46.125079 5437 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a290bd4-9950-4806-bf86-1f02c8b60ea6-tmp-dir\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:38:46.330524 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:46.330488 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-zjbzg" Nov 03 01:38:46.330963 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:46.330799 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-zjbzg" Nov 03 01:38:46.331866 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:46.331847 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-zjbzg" Nov 03 01:38:46.508785 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:46.508683 5437 generic.go:356] "Generic (PLEG): container finished" podID="9a290bd4-9950-4806-bf86-1f02c8b60ea6" containerID="e34f717cfe4652199ac2027d0d735950b648d20662872f890f6818171dfb3ac0" exitCode=0 Nov 03 01:38:46.508785 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:46.508738 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e1793d/e2e-conflict-a-9de9e4b3-kserve-79884b557b-bp2sj" Nov 03 01:38:46.508785 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:46.508759 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e1793d/e2e-conflict-a-9de9e4b3-kserve-79884b557b-bp2sj" event={"ID":"9a290bd4-9950-4806-bf86-1f02c8b60ea6","Type":"ContainerDied","Data":"e34f717cfe4652199ac2027d0d735950b648d20662872f890f6818171dfb3ac0"} Nov 03 01:38:46.509073 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:46.508806 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e1793d/e2e-conflict-a-9de9e4b3-kserve-79884b557b-bp2sj" event={"ID":"9a290bd4-9950-4806-bf86-1f02c8b60ea6","Type":"ContainerDied","Data":"ef0d62f4717b950b63c3fb0c824e398548e882a424f0b8ef87eab1b2002ca68c"} Nov 03 01:38:46.509073 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:46.508826 5437 scope.go:122] "RemoveContainer" containerID="e34f717cfe4652199ac2027d0d735950b648d20662872f890f6818171dfb3ac0" Nov 03 01:38:46.519062 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:46.519036 5437 scope.go:122] "RemoveContainer" containerID="e34f717cfe4652199ac2027d0d735950b648d20662872f890f6818171dfb3ac0" Nov 03 01:38:46.519466 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:38:46.519443 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e34f717cfe4652199ac2027d0d735950b648d20662872f890f6818171dfb3ac0\": container with ID starting with e34f717cfe4652199ac2027d0d735950b648d20662872f890f6818171dfb3ac0 not found: ID does not exist" containerID="e34f717cfe4652199ac2027d0d735950b648d20662872f890f6818171dfb3ac0" Nov 03 01:38:46.519543 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:46.519476 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e34f717cfe4652199ac2027d0d735950b648d20662872f890f6818171dfb3ac0"} err="failed to get container status \"e34f717cfe4652199ac2027d0d735950b648d20662872f890f6818171dfb3ac0\": rpc error: code = NotFound desc = could not find container \"e34f717cfe4652199ac2027d0d735950b648d20662872f890f6818171dfb3ac0\": container with ID starting with e34f717cfe4652199ac2027d0d735950b648d20662872f890f6818171dfb3ac0 not found: ID does not exist" Nov 03 01:38:46.610520 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:46.610477 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-gh4dm"] Nov 03 01:38:46.611205 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:46.611178 5437 cpu_manager.go:414] "containerMap: removing container" podUID="9a290bd4-9950-4806-bf86-1f02c8b60ea6" containerName="main" Nov 03 01:38:46.611205 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:46.611204 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a290bd4-9950-4806-bf86-1f02c8b60ea6" containerName="main" Nov 03 01:38:46.611492 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:46.611423 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a290bd4-9950-4806-bf86-1f02c8b60ea6" containerName="main" Nov 03 01:38:46.616688 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:46.616656 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-gh4dm" Nov 03 01:38:46.696909 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:46.696857 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-gh4dm"] Nov 03 01:38:46.703968 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:46.703919 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-e1793d/e2e-conflict-a-9de9e4b3-kserve-79884b557b-bp2sj"] Nov 03 01:38:46.707833 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:46.707791 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-e1793d/e2e-conflict-a-9de9e4b3-kserve-79884b557b-bp2sj"] Nov 03 01:38:46.733333 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:46.733294 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27d14b9a-3320-4e79-85b6-662d86877310-istiod-ca-cert\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-gh4dm\" (UID: \"27d14b9a-3320-4e79-85b6-662d86877310\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-gh4dm" Nov 03 01:38:46.733541 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:46.733457 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjz8c\" (UniqueName: \"kubernetes.io/projected/27d14b9a-3320-4e79-85b6-662d86877310-kube-api-access-pjz8c\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-gh4dm\" (UID: \"27d14b9a-3320-4e79-85b6-662d86877310\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-gh4dm" Nov 03 01:38:46.733541 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:46.733510 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27d14b9a-3320-4e79-85b6-662d86877310-istio-data\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-gh4dm\" (UID: \"27d14b9a-3320-4e79-85b6-662d86877310\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-gh4dm" Nov 03 01:38:46.733652 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:46.733565 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27d14b9a-3320-4e79-85b6-662d86877310-workload-socket\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-gh4dm\" (UID: \"27d14b9a-3320-4e79-85b6-662d86877310\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-gh4dm" Nov 03 01:38:46.733694 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:46.733652 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27d14b9a-3320-4e79-85b6-662d86877310-istio-token\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-gh4dm\" (UID: \"27d14b9a-3320-4e79-85b6-662d86877310\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-gh4dm" Nov 03 01:38:46.733731 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:46.733710 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27d14b9a-3320-4e79-85b6-662d86877310-workload-certs\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-gh4dm\" (UID: \"27d14b9a-3320-4e79-85b6-662d86877310\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-gh4dm" Nov 03 01:38:46.733825 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:46.733803 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27d14b9a-3320-4e79-85b6-662d86877310-istio-envoy\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-gh4dm\" (UID: \"27d14b9a-3320-4e79-85b6-662d86877310\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-gh4dm" Nov 03 01:38:46.733866 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:46.733847 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27d14b9a-3320-4e79-85b6-662d86877310-credential-socket\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-gh4dm\" (UID: \"27d14b9a-3320-4e79-85b6-662d86877310\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-gh4dm" Nov 03 01:38:46.733927 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:46.733915 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27d14b9a-3320-4e79-85b6-662d86877310-istio-podinfo\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-gh4dm\" (UID: \"27d14b9a-3320-4e79-85b6-662d86877310\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-gh4dm" Nov 03 01:38:46.835281 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:46.835239 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27d14b9a-3320-4e79-85b6-662d86877310-istio-podinfo\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-gh4dm\" (UID: \"27d14b9a-3320-4e79-85b6-662d86877310\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-gh4dm" Nov 03 01:38:46.835554 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:46.835299 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27d14b9a-3320-4e79-85b6-662d86877310-istiod-ca-cert\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-gh4dm\" (UID: \"27d14b9a-3320-4e79-85b6-662d86877310\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-gh4dm" Nov 03 01:38:46.835554 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:46.835348 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjz8c\" (UniqueName: \"kubernetes.io/projected/27d14b9a-3320-4e79-85b6-662d86877310-kube-api-access-pjz8c\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-gh4dm\" (UID: \"27d14b9a-3320-4e79-85b6-662d86877310\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-gh4dm" Nov 03 01:38:46.835554 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:46.835431 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27d14b9a-3320-4e79-85b6-662d86877310-istio-data\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-gh4dm\" (UID: \"27d14b9a-3320-4e79-85b6-662d86877310\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-gh4dm" Nov 03 01:38:46.835729 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:46.835582 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27d14b9a-3320-4e79-85b6-662d86877310-workload-socket\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-gh4dm\" (UID: \"27d14b9a-3320-4e79-85b6-662d86877310\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-gh4dm" Nov 03 01:38:46.835729 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:46.835624 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27d14b9a-3320-4e79-85b6-662d86877310-istio-token\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-gh4dm\" (UID: \"27d14b9a-3320-4e79-85b6-662d86877310\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-gh4dm" Nov 03 01:38:46.835729 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:46.835678 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27d14b9a-3320-4e79-85b6-662d86877310-workload-certs\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-gh4dm\" (UID: \"27d14b9a-3320-4e79-85b6-662d86877310\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-gh4dm" Nov 03 01:38:46.835729 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:46.835715 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27d14b9a-3320-4e79-85b6-662d86877310-istio-envoy\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-gh4dm\" (UID: \"27d14b9a-3320-4e79-85b6-662d86877310\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-gh4dm" Nov 03 01:38:46.835975 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:46.835743 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27d14b9a-3320-4e79-85b6-662d86877310-credential-socket\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-gh4dm\" (UID: \"27d14b9a-3320-4e79-85b6-662d86877310\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-gh4dm" Nov 03 01:38:46.835975 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:46.835945 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27d14b9a-3320-4e79-85b6-662d86877310-workload-socket\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-gh4dm\" (UID: \"27d14b9a-3320-4e79-85b6-662d86877310\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-gh4dm" Nov 03 01:38:46.836092 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:46.835989 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27d14b9a-3320-4e79-85b6-662d86877310-istio-data\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-gh4dm\" (UID: \"27d14b9a-3320-4e79-85b6-662d86877310\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-gh4dm" Nov 03 01:38:46.836092 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:46.836055 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27d14b9a-3320-4e79-85b6-662d86877310-workload-certs\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-gh4dm\" (UID: \"27d14b9a-3320-4e79-85b6-662d86877310\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-gh4dm" Nov 03 01:38:46.836191 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:46.836162 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27d14b9a-3320-4e79-85b6-662d86877310-istiod-ca-cert\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-gh4dm\" (UID: \"27d14b9a-3320-4e79-85b6-662d86877310\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-gh4dm" Nov 03 01:38:46.836236 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:46.836183 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27d14b9a-3320-4e79-85b6-662d86877310-credential-socket\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-gh4dm\" (UID: \"27d14b9a-3320-4e79-85b6-662d86877310\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-gh4dm" Nov 03 01:38:46.838110 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:46.838085 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27d14b9a-3320-4e79-85b6-662d86877310-istio-envoy\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-gh4dm\" (UID: \"27d14b9a-3320-4e79-85b6-662d86877310\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-gh4dm" Nov 03 01:38:46.838340 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:46.838318 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27d14b9a-3320-4e79-85b6-662d86877310-istio-podinfo\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-gh4dm\" (UID: \"27d14b9a-3320-4e79-85b6-662d86877310\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-gh4dm" Nov 03 01:38:46.898680 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:46.898641 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27d14b9a-3320-4e79-85b6-662d86877310-istio-token\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-gh4dm\" (UID: \"27d14b9a-3320-4e79-85b6-662d86877310\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-gh4dm" Nov 03 01:38:46.898921 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:46.898896 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjz8c\" (UniqueName: \"kubernetes.io/projected/27d14b9a-3320-4e79-85b6-662d86877310-kube-api-access-pjz8c\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-gh4dm\" (UID: \"27d14b9a-3320-4e79-85b6-662d86877310\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-gh4dm" Nov 03 01:38:46.934343 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:46.934304 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-gh4dm" Nov 03 01:38:47.357418 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:38:47.357355 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27d14b9a_3320_4e79_85b6_662d86877310.slice/crio-c0a1967e50f7061548eb91240a2ec5f3d7dab900c9b62819b2d774a267e6449b WatchSource:0}: Error finding container c0a1967e50f7061548eb91240a2ec5f3d7dab900c9b62819b2d774a267e6449b: Status 404 returned error can't find the container with id c0a1967e50f7061548eb91240a2ec5f3d7dab900c9b62819b2d774a267e6449b Nov 03 01:38:47.359762 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:47.359730 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 01:38:47.359874 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:47.359811 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 01:38:47.359874 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:47.359855 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 01:38:47.360660 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:47.360636 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-gh4dm"] Nov 03 01:38:47.518420 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:47.518373 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-gh4dm" event={"ID":"27d14b9a-3320-4e79-85b6-662d86877310","Type":"ContainerStarted","Data":"32940d908f0602e5c9546989b63eb6cd41bd35095d5bf1dcea6a5e8d190b5052"} Nov 03 01:38:47.518614 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:47.518433 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-gh4dm" event={"ID":"27d14b9a-3320-4e79-85b6-662d86877310","Type":"ContainerStarted","Data":"c0a1967e50f7061548eb91240a2ec5f3d7dab900c9b62819b2d774a267e6449b"} Nov 03 01:38:47.643597 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:47.643480 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-gh4dm" podStartSLOduration=1.6434648859999998 podStartE2EDuration="1.643464886s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:38:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:38:47.641346459 +0000 UTC m=+1209.670828766" watchObservedRunningTime="2026-11-03 01:38:47.643464886 +0000 UTC m=+1209.672947181" Nov 03 01:38:47.942445 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:47.936469 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-gh4dm" Nov 03 01:38:47.981934 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:47.981903 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:38:48.094123 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:48.094087 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a290bd4-9950-4806-bf86-1f02c8b60ea6" path="/var/lib/kubelet/pods/9a290bd4-9950-4806-bf86-1f02c8b60ea6/volumes" Nov 03 01:38:48.942392 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:48.938446 5437 patch_prober.go:29] interesting pod/e2e-worker-w2-9da431-openshift-default-65c799fd77-gh4dm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 03 01:38:48.942392 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:48.938520 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-gh4dm" podUID="27d14b9a-3320-4e79-85b6-662d86877310" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 03 01:38:49.938870 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:49.938826 5437 patch_prober.go:29] interesting pod/e2e-worker-w2-9da431-openshift-default-65c799fd77-gh4dm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 03 01:38:49.939186 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:49.939153 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-gh4dm" podUID="27d14b9a-3320-4e79-85b6-662d86877310" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 03 01:38:50.944390 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:50.943504 5437 patch_prober.go:29] interesting pod/e2e-worker-w2-9da431-openshift-default-65c799fd77-gh4dm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 03 01:38:50.944390 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:50.943586 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-gh4dm" podUID="27d14b9a-3320-4e79-85b6-662d86877310" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 03 01:38:51.941390 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:51.939807 5437 patch_prober.go:29] interesting pod/e2e-worker-w2-9da431-openshift-default-65c799fd77-gh4dm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 03 01:38:51.941390 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:51.939886 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-gh4dm" podUID="27d14b9a-3320-4e79-85b6-662d86877310" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 03 01:38:52.942863 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:52.942819 5437 patch_prober.go:29] interesting pod/e2e-worker-w2-9da431-openshift-default-65c799fd77-gh4dm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 03 01:38:52.943329 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:52.942889 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-gh4dm" podUID="27d14b9a-3320-4e79-85b6-662d86877310" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 03 01:38:53.305443 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:53.305405 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:38:53.935542 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:53.935509 5437 patch_prober.go:29] interesting pod/e2e-worker-w2-9da431-openshift-default-65c799fd77-gh4dm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 03 01:38:53.935740 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:53.935582 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-gh4dm" podUID="27d14b9a-3320-4e79-85b6-662d86877310" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 03 01:38:55.977583 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:55.977533 5437 patch_prober.go:29] interesting pod/e2e-worker-w2-9da431-openshift-default-65c799fd77-gh4dm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 01:38:55.978104 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:55.977615 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-gh4dm" podUID="27d14b9a-3320-4e79-85b6-662d86877310" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 01:38:56.060920 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:56.060883 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-gh4dm" Nov 03 01:38:56.061242 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:56.061194 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-gh4dm" Nov 03 01:38:56.062143 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:56.062125 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-gh4dm" Nov 03 01:38:58.109277 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:38:58.109235 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:39:02.886572 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:02.886536 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:39:07.988522 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:07.988476 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:39:09.943830 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:09.943793 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-x7znt"] Nov 03 01:39:09.947979 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:09.947957 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-x7znt" Nov 03 01:39:09.985859 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:09.985816 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de0ef434-0294-4dba-8112-317298a2acb5-istiod-ca-cert\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-x7znt\" (UID: \"de0ef434-0294-4dba-8112-317298a2acb5\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-x7znt" Nov 03 01:39:09.986039 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:09.986005 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6lvq\" (UniqueName: \"kubernetes.io/projected/de0ef434-0294-4dba-8112-317298a2acb5-kube-api-access-l6lvq\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-x7znt\" (UID: \"de0ef434-0294-4dba-8112-317298a2acb5\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-x7znt" Nov 03 01:39:09.986099 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:09.986044 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de0ef434-0294-4dba-8112-317298a2acb5-workload-certs\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-x7znt\" (UID: \"de0ef434-0294-4dba-8112-317298a2acb5\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-x7znt" Nov 03 01:39:09.986099 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:09.986085 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de0ef434-0294-4dba-8112-317298a2acb5-workload-socket\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-x7znt\" (UID: \"de0ef434-0294-4dba-8112-317298a2acb5\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-x7znt" Nov 03 01:39:09.986201 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:09.986114 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de0ef434-0294-4dba-8112-317298a2acb5-istio-podinfo\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-x7znt\" (UID: \"de0ef434-0294-4dba-8112-317298a2acb5\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-x7znt" Nov 03 01:39:09.986315 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:09.986297 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de0ef434-0294-4dba-8112-317298a2acb5-istio-data\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-x7znt\" (UID: \"de0ef434-0294-4dba-8112-317298a2acb5\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-x7znt" Nov 03 01:39:09.986409 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:09.986333 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de0ef434-0294-4dba-8112-317298a2acb5-credential-socket\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-x7znt\" (UID: \"de0ef434-0294-4dba-8112-317298a2acb5\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-x7znt" Nov 03 01:39:09.986409 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:09.986391 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de0ef434-0294-4dba-8112-317298a2acb5-istio-envoy\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-x7znt\" (UID: \"de0ef434-0294-4dba-8112-317298a2acb5\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-x7znt" Nov 03 01:39:09.986523 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:09.986419 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de0ef434-0294-4dba-8112-317298a2acb5-istio-token\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-x7znt\" (UID: \"de0ef434-0294-4dba-8112-317298a2acb5\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-x7znt" Nov 03 01:39:10.056874 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:10.056836 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-x7znt"] Nov 03 01:39:10.087299 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:10.087259 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de0ef434-0294-4dba-8112-317298a2acb5-istiod-ca-cert\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-x7znt\" (UID: \"de0ef434-0294-4dba-8112-317298a2acb5\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-x7znt" Nov 03 01:39:10.087516 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:10.087430 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6lvq\" (UniqueName: \"kubernetes.io/projected/de0ef434-0294-4dba-8112-317298a2acb5-kube-api-access-l6lvq\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-x7znt\" (UID: \"de0ef434-0294-4dba-8112-317298a2acb5\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-x7znt" Nov 03 01:39:10.087516 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:10.087471 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de0ef434-0294-4dba-8112-317298a2acb5-workload-certs\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-x7znt\" (UID: \"de0ef434-0294-4dba-8112-317298a2acb5\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-x7znt" Nov 03 01:39:10.087516 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:10.087498 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de0ef434-0294-4dba-8112-317298a2acb5-workload-socket\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-x7znt\" (UID: \"de0ef434-0294-4dba-8112-317298a2acb5\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-x7znt" Nov 03 01:39:10.087647 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:10.087524 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de0ef434-0294-4dba-8112-317298a2acb5-istio-podinfo\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-x7znt\" (UID: \"de0ef434-0294-4dba-8112-317298a2acb5\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-x7znt" Nov 03 01:39:10.087683 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:10.087649 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de0ef434-0294-4dba-8112-317298a2acb5-istio-data\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-x7znt\" (UID: \"de0ef434-0294-4dba-8112-317298a2acb5\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-x7znt" Nov 03 01:39:10.087723 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:10.087676 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de0ef434-0294-4dba-8112-317298a2acb5-credential-socket\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-x7znt\" (UID: \"de0ef434-0294-4dba-8112-317298a2acb5\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-x7znt" Nov 03 01:39:10.087723 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:10.087707 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de0ef434-0294-4dba-8112-317298a2acb5-istio-envoy\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-x7znt\" (UID: \"de0ef434-0294-4dba-8112-317298a2acb5\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-x7znt" Nov 03 01:39:10.087796 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:10.087732 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de0ef434-0294-4dba-8112-317298a2acb5-istio-token\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-x7znt\" (UID: \"de0ef434-0294-4dba-8112-317298a2acb5\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-x7znt" Nov 03 01:39:10.087881 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:10.087854 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de0ef434-0294-4dba-8112-317298a2acb5-workload-certs\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-x7znt\" (UID: \"de0ef434-0294-4dba-8112-317298a2acb5\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-x7znt" Nov 03 01:39:10.088010 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:10.087980 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de0ef434-0294-4dba-8112-317298a2acb5-istio-data\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-x7znt\" (UID: \"de0ef434-0294-4dba-8112-317298a2acb5\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-x7znt" Nov 03 01:39:10.088114 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:10.088095 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de0ef434-0294-4dba-8112-317298a2acb5-istiod-ca-cert\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-x7znt\" (UID: \"de0ef434-0294-4dba-8112-317298a2acb5\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-x7znt" Nov 03 01:39:10.088182 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:10.088123 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de0ef434-0294-4dba-8112-317298a2acb5-credential-socket\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-x7znt\" (UID: \"de0ef434-0294-4dba-8112-317298a2acb5\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-x7znt" Nov 03 01:39:10.088343 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:10.088309 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de0ef434-0294-4dba-8112-317298a2acb5-workload-socket\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-x7znt\" (UID: \"de0ef434-0294-4dba-8112-317298a2acb5\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-x7znt" Nov 03 01:39:10.090471 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:10.090441 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de0ef434-0294-4dba-8112-317298a2acb5-istio-envoy\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-x7znt\" (UID: \"de0ef434-0294-4dba-8112-317298a2acb5\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-x7znt" Nov 03 01:39:10.090643 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:10.090626 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de0ef434-0294-4dba-8112-317298a2acb5-istio-podinfo\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-x7znt\" (UID: \"de0ef434-0294-4dba-8112-317298a2acb5\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-x7znt" Nov 03 01:39:10.150774 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:10.150733 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de0ef434-0294-4dba-8112-317298a2acb5-istio-token\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-x7znt\" (UID: \"de0ef434-0294-4dba-8112-317298a2acb5\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-x7znt" Nov 03 01:39:10.150932 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:10.150789 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6lvq\" (UniqueName: \"kubernetes.io/projected/de0ef434-0294-4dba-8112-317298a2acb5-kube-api-access-l6lvq\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-x7znt\" (UID: \"de0ef434-0294-4dba-8112-317298a2acb5\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-x7znt" Nov 03 01:39:10.262497 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:10.262404 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-x7znt" Nov 03 01:39:10.683654 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:39:10.683619 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde0ef434_0294_4dba_8112_317298a2acb5.slice/crio-d6b192f119ae80339c4c515f0bf3f65a96b36ade3618e7b98ebac9da7bdec974 WatchSource:0}: Error finding container d6b192f119ae80339c4c515f0bf3f65a96b36ade3618e7b98ebac9da7bdec974: Status 404 returned error can't find the container with id d6b192f119ae80339c4c515f0bf3f65a96b36ade3618e7b98ebac9da7bdec974 Nov 03 01:39:10.686389 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:10.686327 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 01:39:10.686472 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:10.686430 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 01:39:10.686472 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:10.686464 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 01:39:10.686885 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:10.686859 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-x7znt"] Nov 03 01:39:10.824484 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:10.824449 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-x7znt" event={"ID":"de0ef434-0294-4dba-8112-317298a2acb5","Type":"ContainerStarted","Data":"4d91470adf4671f55a8a7b9bc5f3866698157cd49eb0b5d735340df9460edcde"} Nov 03 01:39:10.824694 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:10.824511 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-x7znt" event={"ID":"de0ef434-0294-4dba-8112-317298a2acb5","Type":"ContainerStarted","Data":"d6b192f119ae80339c4c515f0bf3f65a96b36ade3618e7b98ebac9da7bdec974"} Nov 03 01:39:10.948698 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:10.948597 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-x7znt" podStartSLOduration=1.94858229 podStartE2EDuration="1.94858229s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:39:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:39:10.947913263 +0000 UTC m=+1232.977395570" watchObservedRunningTime="2026-11-03 01:39:10.94858229 +0000 UTC m=+1232.978064585" Nov 03 01:39:11.263553 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:11.263446 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-x7znt" Nov 03 01:39:11.265461 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:11.265423 5437 patch_prober.go:29] interesting pod/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-x7znt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 03 01:39:11.265608 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:11.265476 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-x7znt" podUID="de0ef434-0294-4dba-8112-317298a2acb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 03 01:39:12.263842 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:12.263806 5437 patch_prober.go:29] interesting pod/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-x7znt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 03 01:39:12.264258 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:12.263862 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-x7znt" podUID="de0ef434-0294-4dba-8112-317298a2acb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 03 01:39:12.816415 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:12.816329 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:39:13.263150 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:13.263065 5437 patch_prober.go:29] interesting pod/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-x7znt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 03 01:39:13.263150 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:13.263118 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-x7znt" podUID="de0ef434-0294-4dba-8112-317298a2acb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 03 01:39:14.061752 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:39:14.061714 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 01:39:14.263262 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:14.263223 5437 patch_prober.go:29] interesting pod/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-x7znt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 03 01:39:14.263458 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:14.263279 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-x7znt" podUID="de0ef434-0294-4dba-8112-317298a2acb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 03 01:39:15.263496 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:15.263452 5437 patch_prober.go:29] interesting pod/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-x7znt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 03 01:39:15.264000 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:15.263519 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-x7znt" podUID="de0ef434-0294-4dba-8112-317298a2acb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 03 01:39:16.263045 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:16.263002 5437 patch_prober.go:29] interesting pod/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-x7znt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 03 01:39:16.263234 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:16.263078 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-x7znt" podUID="de0ef434-0294-4dba-8112-317298a2acb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 03 01:39:17.266418 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:17.264408 5437 patch_prober.go:29] interesting pod/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-x7znt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 03 01:39:17.266418 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:17.264493 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-x7znt" podUID="de0ef434-0294-4dba-8112-317298a2acb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 03 01:39:18.086390 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:18.085751 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:39:19.267233 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:19.267189 5437 patch_prober.go:29] interesting pod/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-x7znt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 01:39:19.267780 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:19.267253 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-x7znt" podUID="de0ef434-0294-4dba-8112-317298a2acb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 01:39:19.999887 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:19.999849 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-x7znt" Nov 03 01:39:20.000324 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:20.000296 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-x7znt" Nov 03 01:39:20.001097 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:20.001076 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-x7znt" Nov 03 01:39:22.905293 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:22.905259 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:39:23.315599 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:23.315559 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-g5ggm"] Nov 03 01:39:23.320152 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:23.320122 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-g5ggm" Nov 03 01:39:23.367292 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:23.367251 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-e1793d/distinct-w1-e1793d-kserve-687855c49-vn8zs"] Nov 03 01:39:23.372446 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:23.372417 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e1793d/distinct-w1-e1793d-kserve-687855c49-vn8zs" Nov 03 01:39:23.415417 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:23.410319 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ab5aefa-9401-4a6d-9168-fff8422de5b3-istio-podinfo\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-g5ggm\" (UID: \"1ab5aefa-9401-4a6d-9168-fff8422de5b3\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-g5ggm" Nov 03 01:39:23.415417 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:23.410417 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ab5aefa-9401-4a6d-9168-fff8422de5b3-istio-data\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-g5ggm\" (UID: \"1ab5aefa-9401-4a6d-9168-fff8422de5b3\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-g5ggm" Nov 03 01:39:23.415417 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:23.410468 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a821ee29-70aa-4cad-87d4-3fb0edd20810-model-cache\") pod \"distinct-w1-e1793d-kserve-687855c49-vn8zs\" (UID: \"a821ee29-70aa-4cad-87d4-3fb0edd20810\") " pod="e2e-models-e2e-worker-w1-e1793d/distinct-w1-e1793d-kserve-687855c49-vn8zs" Nov 03 01:39:23.415417 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:23.410505 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4qrg\" (UniqueName: \"kubernetes.io/projected/a821ee29-70aa-4cad-87d4-3fb0edd20810-kube-api-access-r4qrg\") pod \"distinct-w1-e1793d-kserve-687855c49-vn8zs\" (UID: \"a821ee29-70aa-4cad-87d4-3fb0edd20810\") " pod="e2e-models-e2e-worker-w1-e1793d/distinct-w1-e1793d-kserve-687855c49-vn8zs" Nov 03 01:39:23.415417 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:23.410542 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ab5aefa-9401-4a6d-9168-fff8422de5b3-istio-envoy\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-g5ggm\" (UID: \"1ab5aefa-9401-4a6d-9168-fff8422de5b3\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-g5ggm" Nov 03 01:39:23.415417 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:23.410566 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ab5aefa-9401-4a6d-9168-fff8422de5b3-istio-token\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-g5ggm\" (UID: \"1ab5aefa-9401-4a6d-9168-fff8422de5b3\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-g5ggm" Nov 03 01:39:23.415417 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:23.410602 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a821ee29-70aa-4cad-87d4-3fb0edd20810-tls-certs\") pod \"distinct-w1-e1793d-kserve-687855c49-vn8zs\" (UID: \"a821ee29-70aa-4cad-87d4-3fb0edd20810\") " pod="e2e-models-e2e-worker-w1-e1793d/distinct-w1-e1793d-kserve-687855c49-vn8zs" Nov 03 01:39:23.415417 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:23.410647 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ab5aefa-9401-4a6d-9168-fff8422de5b3-istiod-ca-cert\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-g5ggm\" (UID: \"1ab5aefa-9401-4a6d-9168-fff8422de5b3\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-g5ggm" Nov 03 01:39:23.415417 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:23.410757 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qgf9\" (UniqueName: \"kubernetes.io/projected/1ab5aefa-9401-4a6d-9168-fff8422de5b3-kube-api-access-6qgf9\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-g5ggm\" (UID: \"1ab5aefa-9401-4a6d-9168-fff8422de5b3\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-g5ggm" Nov 03 01:39:23.415417 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:23.410785 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a821ee29-70aa-4cad-87d4-3fb0edd20810-tmp-dir\") pod \"distinct-w1-e1793d-kserve-687855c49-vn8zs\" (UID: \"a821ee29-70aa-4cad-87d4-3fb0edd20810\") " pod="e2e-models-e2e-worker-w1-e1793d/distinct-w1-e1793d-kserve-687855c49-vn8zs" Nov 03 01:39:23.415417 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:23.410828 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ab5aefa-9401-4a6d-9168-fff8422de5b3-workload-socket\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-g5ggm\" (UID: \"1ab5aefa-9401-4a6d-9168-fff8422de5b3\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-g5ggm" Nov 03 01:39:23.415417 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:23.410873 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a821ee29-70aa-4cad-87d4-3fb0edd20810-home\") pod \"distinct-w1-e1793d-kserve-687855c49-vn8zs\" (UID: \"a821ee29-70aa-4cad-87d4-3fb0edd20810\") " pod="e2e-models-e2e-worker-w1-e1793d/distinct-w1-e1793d-kserve-687855c49-vn8zs" Nov 03 01:39:23.415417 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:23.410954 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a821ee29-70aa-4cad-87d4-3fb0edd20810-dshm\") pod \"distinct-w1-e1793d-kserve-687855c49-vn8zs\" (UID: \"a821ee29-70aa-4cad-87d4-3fb0edd20810\") " pod="e2e-models-e2e-worker-w1-e1793d/distinct-w1-e1793d-kserve-687855c49-vn8zs" Nov 03 01:39:23.415417 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:23.410980 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ab5aefa-9401-4a6d-9168-fff8422de5b3-credential-socket\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-g5ggm\" (UID: \"1ab5aefa-9401-4a6d-9168-fff8422de5b3\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-g5ggm" Nov 03 01:39:23.415417 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:23.411012 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ab5aefa-9401-4a6d-9168-fff8422de5b3-workload-certs\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-g5ggm\" (UID: \"1ab5aefa-9401-4a6d-9168-fff8422de5b3\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-g5ggm" Nov 03 01:39:23.416280 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:23.415527 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-g5ggm"] Nov 03 01:39:23.427113 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:23.427083 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-e1793d\"/\"distinct-w1-e1793d-kserve-self-signed-certs\"" Nov 03 01:39:23.449942 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:23.449905 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-e1793d/distinct-w1-e1793d-kserve-687855c49-vn8zs"] Nov 03 01:39:23.501961 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:23.501920 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:39:23.512424 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:23.512382 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a821ee29-70aa-4cad-87d4-3fb0edd20810-model-cache\") pod \"distinct-w1-e1793d-kserve-687855c49-vn8zs\" (UID: \"a821ee29-70aa-4cad-87d4-3fb0edd20810\") " pod="e2e-models-e2e-worker-w1-e1793d/distinct-w1-e1793d-kserve-687855c49-vn8zs" Nov 03 01:39:23.512708 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:23.512444 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4qrg\" (UniqueName: \"kubernetes.io/projected/a821ee29-70aa-4cad-87d4-3fb0edd20810-kube-api-access-r4qrg\") pod \"distinct-w1-e1793d-kserve-687855c49-vn8zs\" (UID: \"a821ee29-70aa-4cad-87d4-3fb0edd20810\") " pod="e2e-models-e2e-worker-w1-e1793d/distinct-w1-e1793d-kserve-687855c49-vn8zs" Nov 03 01:39:23.512708 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:23.512484 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ab5aefa-9401-4a6d-9168-fff8422de5b3-istio-envoy\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-g5ggm\" (UID: \"1ab5aefa-9401-4a6d-9168-fff8422de5b3\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-g5ggm" Nov 03 01:39:23.512708 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:23.512512 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ab5aefa-9401-4a6d-9168-fff8422de5b3-istio-token\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-g5ggm\" (UID: \"1ab5aefa-9401-4a6d-9168-fff8422de5b3\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-g5ggm" Nov 03 01:39:23.512708 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:23.512545 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a821ee29-70aa-4cad-87d4-3fb0edd20810-tls-certs\") pod \"distinct-w1-e1793d-kserve-687855c49-vn8zs\" (UID: \"a821ee29-70aa-4cad-87d4-3fb0edd20810\") " pod="e2e-models-e2e-worker-w1-e1793d/distinct-w1-e1793d-kserve-687855c49-vn8zs" Nov 03 01:39:23.512708 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:23.512586 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ab5aefa-9401-4a6d-9168-fff8422de5b3-istiod-ca-cert\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-g5ggm\" (UID: \"1ab5aefa-9401-4a6d-9168-fff8422de5b3\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-g5ggm" Nov 03 01:39:23.512708 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:23.512684 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qgf9\" (UniqueName: \"kubernetes.io/projected/1ab5aefa-9401-4a6d-9168-fff8422de5b3-kube-api-access-6qgf9\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-g5ggm\" (UID: \"1ab5aefa-9401-4a6d-9168-fff8422de5b3\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-g5ggm" Nov 03 01:39:23.512708 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:23.512709 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a821ee29-70aa-4cad-87d4-3fb0edd20810-tmp-dir\") pod \"distinct-w1-e1793d-kserve-687855c49-vn8zs\" (UID: \"a821ee29-70aa-4cad-87d4-3fb0edd20810\") " pod="e2e-models-e2e-worker-w1-e1793d/distinct-w1-e1793d-kserve-687855c49-vn8zs" Nov 03 01:39:23.513259 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:23.512749 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ab5aefa-9401-4a6d-9168-fff8422de5b3-workload-socket\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-g5ggm\" (UID: \"1ab5aefa-9401-4a6d-9168-fff8422de5b3\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-g5ggm" Nov 03 01:39:23.513259 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:23.512790 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a821ee29-70aa-4cad-87d4-3fb0edd20810-home\") pod \"distinct-w1-e1793d-kserve-687855c49-vn8zs\" (UID: \"a821ee29-70aa-4cad-87d4-3fb0edd20810\") " pod="e2e-models-e2e-worker-w1-e1793d/distinct-w1-e1793d-kserve-687855c49-vn8zs" Nov 03 01:39:23.513259 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:23.512868 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a821ee29-70aa-4cad-87d4-3fb0edd20810-dshm\") pod \"distinct-w1-e1793d-kserve-687855c49-vn8zs\" (UID: \"a821ee29-70aa-4cad-87d4-3fb0edd20810\") " pod="e2e-models-e2e-worker-w1-e1793d/distinct-w1-e1793d-kserve-687855c49-vn8zs" Nov 03 01:39:23.513259 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:23.512891 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ab5aefa-9401-4a6d-9168-fff8422de5b3-credential-socket\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-g5ggm\" (UID: \"1ab5aefa-9401-4a6d-9168-fff8422de5b3\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-g5ggm" Nov 03 01:39:23.513259 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:23.512901 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a821ee29-70aa-4cad-87d4-3fb0edd20810-model-cache\") pod \"distinct-w1-e1793d-kserve-687855c49-vn8zs\" (UID: \"a821ee29-70aa-4cad-87d4-3fb0edd20810\") " pod="e2e-models-e2e-worker-w1-e1793d/distinct-w1-e1793d-kserve-687855c49-vn8zs" Nov 03 01:39:23.513259 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:23.512920 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ab5aefa-9401-4a6d-9168-fff8422de5b3-workload-certs\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-g5ggm\" (UID: \"1ab5aefa-9401-4a6d-9168-fff8422de5b3\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-g5ggm" Nov 03 01:39:23.513259 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:23.513013 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ab5aefa-9401-4a6d-9168-fff8422de5b3-istio-podinfo\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-g5ggm\" (UID: \"1ab5aefa-9401-4a6d-9168-fff8422de5b3\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-g5ggm" Nov 03 01:39:23.513259 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:23.513068 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ab5aefa-9401-4a6d-9168-fff8422de5b3-istio-data\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-g5ggm\" (UID: \"1ab5aefa-9401-4a6d-9168-fff8422de5b3\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-g5ggm" Nov 03 01:39:23.513799 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:23.513556 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ab5aefa-9401-4a6d-9168-fff8422de5b3-istio-data\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-g5ggm\" (UID: \"1ab5aefa-9401-4a6d-9168-fff8422de5b3\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-g5ggm" Nov 03 01:39:23.513863 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:23.513820 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a821ee29-70aa-4cad-87d4-3fb0edd20810-tmp-dir\") pod \"distinct-w1-e1793d-kserve-687855c49-vn8zs\" (UID: \"a821ee29-70aa-4cad-87d4-3fb0edd20810\") " pod="e2e-models-e2e-worker-w1-e1793d/distinct-w1-e1793d-kserve-687855c49-vn8zs" Nov 03 01:39:23.514107 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:23.514068 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ab5aefa-9401-4a6d-9168-fff8422de5b3-workload-socket\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-g5ggm\" (UID: \"1ab5aefa-9401-4a6d-9168-fff8422de5b3\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-g5ggm" Nov 03 01:39:23.514413 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:23.514335 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ab5aefa-9401-4a6d-9168-fff8422de5b3-credential-socket\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-g5ggm\" (UID: \"1ab5aefa-9401-4a6d-9168-fff8422de5b3\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-g5ggm" Nov 03 01:39:23.514518 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:23.514429 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ab5aefa-9401-4a6d-9168-fff8422de5b3-workload-certs\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-g5ggm\" (UID: \"1ab5aefa-9401-4a6d-9168-fff8422de5b3\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-g5ggm" Nov 03 01:39:23.514810 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:23.514782 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a821ee29-70aa-4cad-87d4-3fb0edd20810-home\") pod \"distinct-w1-e1793d-kserve-687855c49-vn8zs\" (UID: \"a821ee29-70aa-4cad-87d4-3fb0edd20810\") " pod="e2e-models-e2e-worker-w1-e1793d/distinct-w1-e1793d-kserve-687855c49-vn8zs" Nov 03 01:39:23.515215 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:23.515187 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ab5aefa-9401-4a6d-9168-fff8422de5b3-istiod-ca-cert\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-g5ggm\" (UID: \"1ab5aefa-9401-4a6d-9168-fff8422de5b3\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-g5ggm" Nov 03 01:39:23.516013 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:23.515975 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ab5aefa-9401-4a6d-9168-fff8422de5b3-istio-envoy\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-g5ggm\" (UID: \"1ab5aefa-9401-4a6d-9168-fff8422de5b3\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-g5ggm" Nov 03 01:39:23.516227 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:23.516202 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a821ee29-70aa-4cad-87d4-3fb0edd20810-dshm\") pod \"distinct-w1-e1793d-kserve-687855c49-vn8zs\" (UID: \"a821ee29-70aa-4cad-87d4-3fb0edd20810\") " pod="e2e-models-e2e-worker-w1-e1793d/distinct-w1-e1793d-kserve-687855c49-vn8zs" Nov 03 01:39:23.516293 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:23.516230 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a821ee29-70aa-4cad-87d4-3fb0edd20810-tls-certs\") pod \"distinct-w1-e1793d-kserve-687855c49-vn8zs\" (UID: \"a821ee29-70aa-4cad-87d4-3fb0edd20810\") " pod="e2e-models-e2e-worker-w1-e1793d/distinct-w1-e1793d-kserve-687855c49-vn8zs" Nov 03 01:39:23.516615 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:23.516599 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ab5aefa-9401-4a6d-9168-fff8422de5b3-istio-podinfo\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-g5ggm\" (UID: \"1ab5aefa-9401-4a6d-9168-fff8422de5b3\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-g5ggm" Nov 03 01:39:23.576909 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:23.576762 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4qrg\" (UniqueName: \"kubernetes.io/projected/a821ee29-70aa-4cad-87d4-3fb0edd20810-kube-api-access-r4qrg\") pod \"distinct-w1-e1793d-kserve-687855c49-vn8zs\" (UID: \"a821ee29-70aa-4cad-87d4-3fb0edd20810\") " pod="e2e-models-e2e-worker-w1-e1793d/distinct-w1-e1793d-kserve-687855c49-vn8zs" Nov 03 01:39:23.577525 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:23.577419 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ab5aefa-9401-4a6d-9168-fff8422de5b3-istio-token\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-g5ggm\" (UID: \"1ab5aefa-9401-4a6d-9168-fff8422de5b3\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-g5ggm" Nov 03 01:39:23.578092 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:23.578062 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qgf9\" (UniqueName: \"kubernetes.io/projected/1ab5aefa-9401-4a6d-9168-fff8422de5b3-kube-api-access-6qgf9\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-g5ggm\" (UID: \"1ab5aefa-9401-4a6d-9168-fff8422de5b3\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-g5ggm" Nov 03 01:39:23.635067 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:23.635029 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-g5ggm" Nov 03 01:39:23.688172 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:23.688131 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e1793d/distinct-w1-e1793d-kserve-687855c49-vn8zs" Nov 03 01:39:24.063154 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:39:24.063112 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ab5aefa_9401_4a6d_9168_fff8422de5b3.slice/crio-cf91bc23675ef02df18f80fdc041f9236279fd74782273ce9f6acdaca197d891 WatchSource:0}: Error finding container cf91bc23675ef02df18f80fdc041f9236279fd74782273ce9f6acdaca197d891: Status 404 returned error can't find the container with id cf91bc23675ef02df18f80fdc041f9236279fd74782273ce9f6acdaca197d891 Nov 03 01:39:24.065951 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:24.065914 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 01:39:24.066065 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:24.066003 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 01:39:24.066065 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:24.066052 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 01:39:24.066528 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:24.066504 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-g5ggm"] Nov 03 01:39:24.109072 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:39:24.109034 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda821ee29_70aa_4cad_87d4_3fb0edd20810.slice/crio-12ee961752075d9ffa31c587c0fe9feba341083be1b1fb5967e8a73c7c7d3ec0 WatchSource:0}: Error finding container 12ee961752075d9ffa31c587c0fe9feba341083be1b1fb5967e8a73c7c7d3ec0: Status 404 returned error can't find the container with id 12ee961752075d9ffa31c587c0fe9feba341083be1b1fb5967e8a73c7c7d3ec0 Nov 03 01:39:24.112711 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:24.112688 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-e1793d/distinct-w1-e1793d-kserve-687855c49-vn8zs"] Nov 03 01:39:24.972498 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:24.972460 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-g5ggm" event={"ID":"1ab5aefa-9401-4a6d-9168-fff8422de5b3","Type":"ContainerStarted","Data":"d8ac94044857ee28411f971121094fc30c570e3c75a574878566c5931896bdfa"} Nov 03 01:39:24.972726 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:24.972523 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-g5ggm" event={"ID":"1ab5aefa-9401-4a6d-9168-fff8422de5b3","Type":"ContainerStarted","Data":"cf91bc23675ef02df18f80fdc041f9236279fd74782273ce9f6acdaca197d891"} Nov 03 01:39:24.974105 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:24.974074 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e1793d/distinct-w1-e1793d-kserve-687855c49-vn8zs" event={"ID":"a821ee29-70aa-4cad-87d4-3fb0edd20810","Type":"ContainerStarted","Data":"1ae7738e8ab100813db6356624907d891521f65a24c45110960cd0a59c93a7fb"} Nov 03 01:39:24.974221 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:24.974113 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e1793d/distinct-w1-e1793d-kserve-687855c49-vn8zs" event={"ID":"a821ee29-70aa-4cad-87d4-3fb0edd20810","Type":"ContainerStarted","Data":"12ee961752075d9ffa31c587c0fe9feba341083be1b1fb5967e8a73c7c7d3ec0"} Nov 03 01:39:25.092327 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:25.092292 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-ltbdx"] Nov 03 01:39:25.096297 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:25.096279 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-ltbdx" Nov 03 01:39:25.108015 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:25.107969 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-g5ggm" podStartSLOduration=2.107957273 podStartE2EDuration="2.107957273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:39:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:39:25.107650801 +0000 UTC m=+1247.137133095" watchObservedRunningTime="2026-11-03 01:39:25.107957273 +0000 UTC m=+1247.137439568" Nov 03 01:39:25.151338 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:25.151290 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18705810-dd2e-41bc-886a-f8ec91844c8f-workload-certs\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-ltbdx\" (UID: \"18705810-dd2e-41bc-886a-f8ec91844c8f\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-ltbdx" Nov 03 01:39:25.151570 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:25.151526 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgqxs\" (UniqueName: \"kubernetes.io/projected/18705810-dd2e-41bc-886a-f8ec91844c8f-kube-api-access-zgqxs\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-ltbdx\" (UID: \"18705810-dd2e-41bc-886a-f8ec91844c8f\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-ltbdx" Nov 03 01:39:25.151651 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:25.151632 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18705810-dd2e-41bc-886a-f8ec91844c8f-workload-socket\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-ltbdx\" (UID: \"18705810-dd2e-41bc-886a-f8ec91844c8f\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-ltbdx" Nov 03 01:39:25.151827 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:25.151775 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18705810-dd2e-41bc-886a-f8ec91844c8f-istio-podinfo\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-ltbdx\" (UID: \"18705810-dd2e-41bc-886a-f8ec91844c8f\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-ltbdx" Nov 03 01:39:25.152340 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:25.152063 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-ltbdx"] Nov 03 01:39:25.152340 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:25.152159 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18705810-dd2e-41bc-886a-f8ec91844c8f-istiod-ca-cert\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-ltbdx\" (UID: \"18705810-dd2e-41bc-886a-f8ec91844c8f\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-ltbdx" Nov 03 01:39:25.152340 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:25.152306 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18705810-dd2e-41bc-886a-f8ec91844c8f-credential-socket\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-ltbdx\" (UID: \"18705810-dd2e-41bc-886a-f8ec91844c8f\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-ltbdx" Nov 03 01:39:25.152340 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:25.152333 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18705810-dd2e-41bc-886a-f8ec91844c8f-istio-data\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-ltbdx\" (UID: \"18705810-dd2e-41bc-886a-f8ec91844c8f\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-ltbdx" Nov 03 01:39:25.152643 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:25.152379 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18705810-dd2e-41bc-886a-f8ec91844c8f-istio-token\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-ltbdx\" (UID: \"18705810-dd2e-41bc-886a-f8ec91844c8f\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-ltbdx" Nov 03 01:39:25.152643 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:25.152443 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18705810-dd2e-41bc-886a-f8ec91844c8f-istio-envoy\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-ltbdx\" (UID: \"18705810-dd2e-41bc-886a-f8ec91844c8f\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-ltbdx" Nov 03 01:39:25.253612 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:25.253523 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18705810-dd2e-41bc-886a-f8ec91844c8f-istio-podinfo\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-ltbdx\" (UID: \"18705810-dd2e-41bc-886a-f8ec91844c8f\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-ltbdx" Nov 03 01:39:25.253612 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:25.253562 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18705810-dd2e-41bc-886a-f8ec91844c8f-istiod-ca-cert\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-ltbdx\" (UID: \"18705810-dd2e-41bc-886a-f8ec91844c8f\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-ltbdx" Nov 03 01:39:25.253612 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:25.253614 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18705810-dd2e-41bc-886a-f8ec91844c8f-credential-socket\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-ltbdx\" (UID: \"18705810-dd2e-41bc-886a-f8ec91844c8f\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-ltbdx" Nov 03 01:39:25.253894 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:25.253634 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18705810-dd2e-41bc-886a-f8ec91844c8f-istio-data\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-ltbdx\" (UID: \"18705810-dd2e-41bc-886a-f8ec91844c8f\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-ltbdx" Nov 03 01:39:25.253894 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:25.253654 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18705810-dd2e-41bc-886a-f8ec91844c8f-istio-token\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-ltbdx\" (UID: \"18705810-dd2e-41bc-886a-f8ec91844c8f\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-ltbdx" Nov 03 01:39:25.253894 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:25.253669 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18705810-dd2e-41bc-886a-f8ec91844c8f-istio-envoy\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-ltbdx\" (UID: \"18705810-dd2e-41bc-886a-f8ec91844c8f\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-ltbdx" Nov 03 01:39:25.253894 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:25.253731 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18705810-dd2e-41bc-886a-f8ec91844c8f-workload-certs\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-ltbdx\" (UID: \"18705810-dd2e-41bc-886a-f8ec91844c8f\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-ltbdx" Nov 03 01:39:25.253894 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:25.253795 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgqxs\" (UniqueName: \"kubernetes.io/projected/18705810-dd2e-41bc-886a-f8ec91844c8f-kube-api-access-zgqxs\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-ltbdx\" (UID: \"18705810-dd2e-41bc-886a-f8ec91844c8f\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-ltbdx" Nov 03 01:39:25.253894 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:25.253838 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18705810-dd2e-41bc-886a-f8ec91844c8f-workload-socket\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-ltbdx\" (UID: \"18705810-dd2e-41bc-886a-f8ec91844c8f\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-ltbdx" Nov 03 01:39:25.254198 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:25.254099 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18705810-dd2e-41bc-886a-f8ec91844c8f-istio-data\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-ltbdx\" (UID: \"18705810-dd2e-41bc-886a-f8ec91844c8f\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-ltbdx" Nov 03 01:39:25.254284 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:25.254255 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18705810-dd2e-41bc-886a-f8ec91844c8f-workload-socket\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-ltbdx\" (UID: \"18705810-dd2e-41bc-886a-f8ec91844c8f\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-ltbdx" Nov 03 01:39:25.254350 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:25.254283 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18705810-dd2e-41bc-886a-f8ec91844c8f-credential-socket\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-ltbdx\" (UID: \"18705810-dd2e-41bc-886a-f8ec91844c8f\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-ltbdx" Nov 03 01:39:25.254350 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:25.254307 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18705810-dd2e-41bc-886a-f8ec91844c8f-workload-certs\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-ltbdx\" (UID: \"18705810-dd2e-41bc-886a-f8ec91844c8f\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-ltbdx" Nov 03 01:39:25.254986 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:25.254963 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18705810-dd2e-41bc-886a-f8ec91844c8f-istiod-ca-cert\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-ltbdx\" (UID: \"18705810-dd2e-41bc-886a-f8ec91844c8f\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-ltbdx" Nov 03 01:39:25.256226 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:25.256199 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18705810-dd2e-41bc-886a-f8ec91844c8f-istio-envoy\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-ltbdx\" (UID: \"18705810-dd2e-41bc-886a-f8ec91844c8f\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-ltbdx" Nov 03 01:39:25.256449 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:25.256427 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18705810-dd2e-41bc-886a-f8ec91844c8f-istio-podinfo\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-ltbdx\" (UID: \"18705810-dd2e-41bc-886a-f8ec91844c8f\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-ltbdx" Nov 03 01:39:25.316925 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:25.316893 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgqxs\" (UniqueName: \"kubernetes.io/projected/18705810-dd2e-41bc-886a-f8ec91844c8f-kube-api-access-zgqxs\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-ltbdx\" (UID: \"18705810-dd2e-41bc-886a-f8ec91844c8f\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-ltbdx" Nov 03 01:39:25.317063 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:25.316899 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18705810-dd2e-41bc-886a-f8ec91844c8f-istio-token\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-ltbdx\" (UID: \"18705810-dd2e-41bc-886a-f8ec91844c8f\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-ltbdx" Nov 03 01:39:25.362685 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:25.362608 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-e1793d/distinct-w1-e1793d-kserve-687855c49-vn8zs" podStartSLOduration=2.36259042 podStartE2EDuration="2.36259042s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:39:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:39:25.361923188 +0000 UTC m=+1247.391405484" watchObservedRunningTime="2026-11-03 01:39:25.36259042 +0000 UTC m=+1247.392072715" Nov 03 01:39:25.410679 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:25.410650 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-ltbdx" Nov 03 01:39:25.642561 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:25.642516 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-g5ggm" Nov 03 01:39:25.643423 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:25.643352 5437 patch_prober.go:29] interesting pod/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-g5ggm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 03 01:39:25.643513 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:25.643462 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-g5ggm" podUID="1ab5aefa-9401-4a6d-9168-fff8422de5b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 03 01:39:25.689152 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:25.689124 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e1793d/distinct-w1-e1793d-kserve-687855c49-vn8zs" Nov 03 01:39:25.864672 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:39:25.864640 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18705810_dd2e_41bc_886a_f8ec91844c8f.slice/crio-3b0123b7bd2dab840390c0cfa968728a427314a90fc0184eae76c84fc3ac6d7b WatchSource:0}: Error finding container 3b0123b7bd2dab840390c0cfa968728a427314a90fc0184eae76c84fc3ac6d7b: Status 404 returned error can't find the container with id 3b0123b7bd2dab840390c0cfa968728a427314a90fc0184eae76c84fc3ac6d7b Nov 03 01:39:25.867459 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:25.867425 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 01:39:25.867564 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:25.867507 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 01:39:25.867564 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:25.867553 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 01:39:25.868925 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:25.868892 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-ltbdx"] Nov 03 01:39:25.982467 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:25.982192 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-ltbdx" event={"ID":"18705810-dd2e-41bc-886a-f8ec91844c8f","Type":"ContainerStarted","Data":"828b378979be082d469c55917aa81a591296ba654f2dd4d6ebf988c99b0368cf"} Nov 03 01:39:25.982467 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:25.982350 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-ltbdx" event={"ID":"18705810-dd2e-41bc-886a-f8ec91844c8f","Type":"ContainerStarted","Data":"3b0123b7bd2dab840390c0cfa968728a427314a90fc0184eae76c84fc3ac6d7b"} Nov 03 01:39:26.141459 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:26.133562 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-ltbdx" podStartSLOduration=1.133543048 podStartE2EDuration="1.133543048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:39:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:39:26.130881167 +0000 UTC m=+1248.160363464" watchObservedRunningTime="2026-11-03 01:39:26.133543048 +0000 UTC m=+1248.163025343" Nov 03 01:39:26.411658 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:26.411566 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-ltbdx" Nov 03 01:39:26.412974 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:26.412939 5437 patch_prober.go:29] interesting pod/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-ltbdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 03 01:39:26.413118 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:26.412997 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-ltbdx" podUID="18705810-dd2e-41bc-886a-f8ec91844c8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 03 01:39:26.637391 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:26.635913 5437 patch_prober.go:29] interesting pod/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-g5ggm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 03 01:39:26.637391 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:26.635979 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-g5ggm" podUID="1ab5aefa-9401-4a6d-9168-fff8422de5b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 03 01:39:27.411603 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:27.411568 5437 patch_prober.go:29] interesting pod/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-ltbdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 03 01:39:27.411986 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:27.411626 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-ltbdx" podUID="18705810-dd2e-41bc-886a-f8ec91844c8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 03 01:39:27.638889 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:27.638847 5437 patch_prober.go:29] interesting pod/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-g5ggm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 03 01:39:27.639212 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:27.639173 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-g5ggm" podUID="1ab5aefa-9401-4a6d-9168-fff8422de5b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 03 01:39:27.882023 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:27.881951 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:39:28.411733 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:28.411698 5437 patch_prober.go:29] interesting pod/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-ltbdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 03 01:39:28.412134 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:28.411753 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-ltbdx" podUID="18705810-dd2e-41bc-886a-f8ec91844c8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 03 01:39:28.640392 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:28.638848 5437 patch_prober.go:29] interesting pod/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-g5ggm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 03 01:39:28.640392 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:28.638917 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-g5ggm" podUID="1ab5aefa-9401-4a6d-9168-fff8422de5b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 03 01:39:29.411514 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:29.411480 5437 patch_prober.go:29] interesting pod/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-ltbdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 03 01:39:29.411786 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:29.411752 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-ltbdx" podUID="18705810-dd2e-41bc-886a-f8ec91844c8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 03 01:39:29.643401 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:29.640897 5437 patch_prober.go:29] interesting pod/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-g5ggm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 03 01:39:29.643401 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:29.640966 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-g5ggm" podUID="1ab5aefa-9401-4a6d-9168-fff8422de5b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 03 01:39:30.411425 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:30.411391 5437 patch_prober.go:29] interesting pod/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-ltbdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 03 01:39:30.411551 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:30.411454 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-ltbdx" podUID="18705810-dd2e-41bc-886a-f8ec91844c8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 03 01:39:30.642881 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:30.642843 5437 patch_prober.go:29] interesting pod/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-g5ggm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 03 01:39:30.643409 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:30.642926 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-g5ggm" podUID="1ab5aefa-9401-4a6d-9168-fff8422de5b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 03 01:39:31.411777 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:31.411736 5437 patch_prober.go:29] interesting pod/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-ltbdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 03 01:39:31.412013 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:31.411818 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-ltbdx" podUID="18705810-dd2e-41bc-886a-f8ec91844c8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 03 01:39:31.712074 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:31.711978 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-g5ggm" Nov 03 01:39:31.712774 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:31.712745 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-g5ggm" Nov 03 01:39:31.713587 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:31.713568 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-g5ggm" Nov 03 01:39:32.412246 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:32.412209 5437 patch_prober.go:29] interesting pod/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-ltbdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 03 01:39:32.412483 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:32.412290 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-ltbdx" podUID="18705810-dd2e-41bc-886a-f8ec91844c8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 03 01:39:32.798343 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:32.798306 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:39:33.415838 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:33.415805 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-ltbdx" Nov 03 01:39:33.416091 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:33.416069 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-ltbdx" Nov 03 01:39:33.416762 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:33.416736 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-ltbdx" Nov 03 01:39:33.688703 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:33.688612 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-e1793d/distinct-w1-e1793d-kserve-687855c49-vn8zs" Nov 03 01:39:33.701391 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:33.701345 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-e1793d/distinct-w1-e1793d-kserve-687855c49-vn8zs" Nov 03 01:39:34.060746 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:34.060712 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e1793d/distinct-w1-e1793d-kserve-687855c49-vn8zs" Nov 03 01:39:36.539401 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:36.524459 5437 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-0bf34a/simulator-w4-0bf34a-kserve-6bc96559d9-gkdwl" podUID="4cfca4a1-0da6-4637-bc48-8e65776b9ca9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:39:36.539401 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:36.525837 5437 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l87z7" podUID="03a82c52-52db-44b7-bf71-64ec0c963054" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:39:36.803393 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:36.799805 5437 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-41aa5f/simulator-w3-41aa5f-kserve-586cfbcdd4-gsmfk" podUID="8a87f809-0b26-480e-bb37-ff234a40c5c5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Nov 03 01:39:36.955396 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:36.954467 5437 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0bf34a/simulator-w4-0bf34a-kserve-6bc96559d9-gkdwl" podUID="4cfca4a1-0da6-4637-bc48-8e65776b9ca9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:39:37.043395 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:37.039470 5437 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-41aa5f/premium-w3-41aa5f-kserve-5fb94fd5fc-gmtkb" podUID="c46dca02-6036-46de-b0f3-df796d231f62" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:39:37.052391 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:37.051491 5437 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e1793d/distinct-w1-e1793d-kserve-687855c49-vn8zs" podUID="a821ee29-70aa-4cad-87d4-3fb0edd20810" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:39:37.265452 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:37.257514 5437 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e1793d/simulator-w1-e1793d-kserve-7595dfc6b7-dbbgf" podUID="f28ccf15-b92d-400c-9204-69b66279d7d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:39:37.605040 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:37.591203 5437 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l87z7" podUID="03a82c52-52db-44b7-bf71-64ec0c963054" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 01:39:37.605040 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:37.603380 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l87z7" Nov 03 01:39:37.818401 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:37.808450 5437 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-41aa5f/simulator-w3-41aa5f-kserve-586cfbcdd4-gsmfk" podUID="8a87f809-0b26-480e-bb37-ff234a40c5c5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:39:37.818401 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:37.809071 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-41aa5f/simulator-w3-41aa5f-kserve-586cfbcdd4-gsmfk" Nov 03 01:39:37.964384 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:37.964280 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l87z7" Nov 03 01:39:37.965565 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:37.965531 5437 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0bf34a/simulator-w4-0bf34a-kserve-6bc96559d9-gkdwl" podUID="4cfca4a1-0da6-4637-bc48-8e65776b9ca9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:39:37.966306 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:37.966286 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0bf34a/simulator-w4-0bf34a-kserve-6bc96559d9-gkdwl" Nov 03 01:39:38.025513 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:38.025486 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-41aa5f/simulator-w3-41aa5f-kserve-586cfbcdd4-gsmfk" Nov 03 01:39:38.074549 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:38.074499 5437 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-41aa5f/premium-w3-41aa5f-kserve-5fb94fd5fc-gmtkb" podUID="c46dca02-6036-46de-b0f3-df796d231f62" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Nov 03 01:39:38.078038 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:38.078008 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-41aa5f/premium-w3-41aa5f-kserve-5fb94fd5fc-gmtkb" Nov 03 01:39:38.099118 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:38.079431 5437 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e1793d/distinct-w1-e1793d-kserve-687855c49-vn8zs" podUID="a821ee29-70aa-4cad-87d4-3fb0edd20810" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:39:38.099118 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:38.079774 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e1793d/distinct-w1-e1793d-kserve-687855c49-vn8zs" Nov 03 01:39:38.249511 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:38.249433 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e1793d/distinct-w1-e1793d-kserve-687855c49-vn8zs" Nov 03 01:39:38.249730 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:38.249718 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0bf34a/simulator-w4-0bf34a-kserve-6bc96559d9-gkdwl" Nov 03 01:39:38.249853 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:38.249843 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-41aa5f/premium-w3-41aa5f-kserve-5fb94fd5fc-gmtkb" Nov 03 01:39:38.825282 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:38.825234 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:39:39.663273 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:39.663240 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:39:39.876359 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:39.876323 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:39:42.745232 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:42.745189 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:39:43.750538 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:43.750495 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:39:44.036040 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:39:44.035949 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 01:39:48.564930 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:48.564849 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:39:53.315445 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:53.315407 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:39:53.533481 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:53.533442 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:39:58.188914 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:39:58.188849 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:03.113120 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:03.113077 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:03.782965 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:03.782926 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:03.973611 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:03.973532 5437 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l87z7" podUID="03a82c52-52db-44b7-bf71-64ec0c963054" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": context deadline exceeded" Nov 03 01:40:04.037805 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:04.037689 5437 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-41aa5f/simulator-w3-41aa5f-kserve-586cfbcdd4-gsmfk" podUID="8a87f809-0b26-480e-bb37-ff234a40c5c5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Nov 03 01:40:04.114441 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:04.114337 5437 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0bf34a/simulator-w4-0bf34a-kserve-6bc96559d9-gkdwl" podUID="4cfca4a1-0da6-4637-bc48-8e65776b9ca9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:40:04.115991 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:04.115958 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:04.116751 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:04.116717 5437 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e1793d/distinct-w1-e1793d-kserve-687855c49-vn8zs" podUID="a821ee29-70aa-4cad-87d4-3fb0edd20810" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:40:04.239757 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:04.239709 5437 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-41aa5f/premium-w3-41aa5f-kserve-5fb94fd5fc-gmtkb" podUID="c46dca02-6036-46de-b0f3-df796d231f62" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:40:04.245498 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:04.245460 5437 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e1793d/simulator-w1-e1793d-kserve-7595dfc6b7-dbbgf" podUID="f28ccf15-b92d-400c-9204-69b66279d7d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Nov 03 01:40:04.491400 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:04.484990 5437 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l87z7" podUID="03a82c52-52db-44b7-bf71-64ec0c963054" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/health\": context deadline exceeded" Nov 03 01:40:04.694823 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:04.694752 5437 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-e1793d/distinct-w1-e1793d-kserve-687855c49-vn8zs" podUID="a821ee29-70aa-4cad-87d4-3fb0edd20810" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:40:05.078411 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:04.995582 5437 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l87z7" podUID="03a82c52-52db-44b7-bf71-64ec0c963054" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:40:05.078411 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:04.996424 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l87z7" Nov 03 01:40:05.078411 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:05.045483 5437 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-41aa5f/simulator-w3-41aa5f-kserve-586cfbcdd4-gsmfk" podUID="8a87f809-0b26-480e-bb37-ff234a40c5c5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:40:05.078411 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:05.045958 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-41aa5f/simulator-w3-41aa5f-kserve-586cfbcdd4-gsmfk" Nov 03 01:40:05.140410 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:05.119516 5437 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e1793d/distinct-w1-e1793d-kserve-687855c49-vn8zs" podUID="a821ee29-70aa-4cad-87d4-3fb0edd20810" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:40:05.140410 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:05.119957 5437 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0bf34a/simulator-w4-0bf34a-kserve-6bc96559d9-gkdwl" podUID="4cfca4a1-0da6-4637-bc48-8e65776b9ca9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Nov 03 01:40:05.140410 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:05.120306 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e1793d/distinct-w1-e1793d-kserve-687855c49-vn8zs" Nov 03 01:40:05.140410 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:05.120387 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0bf34a/simulator-w4-0bf34a-kserve-6bc96559d9-gkdwl" Nov 03 01:40:05.246405 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:05.244836 5437 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-41aa5f/premium-w3-41aa5f-kserve-5fb94fd5fc-gmtkb" podUID="c46dca02-6036-46de-b0f3-df796d231f62" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:40:05.246405 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:05.245393 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-41aa5f/premium-w3-41aa5f-kserve-5fb94fd5fc-gmtkb" Nov 03 01:40:05.268449 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:05.246774 5437 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e1793d/simulator-w1-e1793d-kserve-7595dfc6b7-dbbgf" podUID="f28ccf15-b92d-400c-9204-69b66279d7d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:40:05.268449 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:05.247187 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e1793d/simulator-w1-e1793d-kserve-7595dfc6b7-dbbgf" Nov 03 01:40:06.059821 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:06.059765 5437 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l87z7" podUID="03a82c52-52db-44b7-bf71-64ec0c963054" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:40:06.104391 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:06.080487 5437 patch_prober.go:29] interesting pod/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-x7znt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 01:40:06.104736 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:06.104693 5437 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-x7znt" podUID="de0ef434-0294-4dba-8112-317298a2acb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 01:40:06.130401 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:06.117241 5437 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-41aa5f/simulator-w3-41aa5f-kserve-586cfbcdd4-gsmfk" podUID="8a87f809-0b26-480e-bb37-ff234a40c5c5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Nov 03 01:40:06.273391 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:06.248391 5437 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e1793d/simulator-w1-e1793d-kserve-7595dfc6b7-dbbgf" podUID="f28ccf15-b92d-400c-9204-69b66279d7d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:40:06.276598 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:06.251531 5437 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-41aa5f/premium-w3-41aa5f-kserve-5fb94fd5fc-gmtkb" podUID="c46dca02-6036-46de-b0f3-df796d231f62" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:40:06.396450 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:06.358456 5437 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e1793d/distinct-w1-e1793d-kserve-687855c49-vn8zs" podUID="a821ee29-70aa-4cad-87d4-3fb0edd20810" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded" Nov 03 01:40:06.445145 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:06.368428 5437 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0bf34a/simulator-w4-0bf34a-kserve-6bc96559d9-gkdwl" podUID="4cfca4a1-0da6-4637-bc48-8e65776b9ca9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Nov 03 01:40:06.521500 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:06.521448 5437 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-0bf34a/simulator-w4-0bf34a-kserve-6bc96559d9-gkdwl" podUID="4cfca4a1-0da6-4637-bc48-8e65776b9ca9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:40:07.013390 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:07.000894 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-41aa5f/simulator-w3-41aa5f-kserve-586cfbcdd4-gsmfk" Nov 03 01:40:07.065044 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:07.063539 5437 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l87z7" podUID="03a82c52-52db-44b7-bf71-64ec0c963054" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:40:07.086588 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:07.085337 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e1793d/simulator-w1-e1793d-kserve-7595dfc6b7-dbbgf" Nov 03 01:40:07.249984 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:07.249633 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-41aa5f/premium-w3-41aa5f-kserve-5fb94fd5fc-gmtkb" Nov 03 01:40:07.279401 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:07.279278 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0bf34a/simulator-w4-0bf34a-kserve-6bc96559d9-gkdwl" Nov 03 01:40:07.309261 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:07.308349 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l87z7" Nov 03 01:40:07.313757 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:07.313712 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e1793d/distinct-w1-e1793d-kserve-687855c49-vn8zs" Nov 03 01:40:08.806841 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:08.806798 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-jxv68"] Nov 03 01:40:08.811251 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:08.811228 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-jxv68" Nov 03 01:40:08.868114 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:08.868074 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-jxv68"] Nov 03 01:40:08.873211 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:08.873172 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f118c89-c5d9-4939-b609-d3c173b27ae5-istio-envoy\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-jxv68\" (UID: \"5f118c89-c5d9-4939-b609-d3c173b27ae5\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-jxv68" Nov 03 01:40:08.873340 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:08.873320 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f118c89-c5d9-4939-b609-d3c173b27ae5-istio-podinfo\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-jxv68\" (UID: \"5f118c89-c5d9-4939-b609-d3c173b27ae5\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-jxv68" Nov 03 01:40:08.873464 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:08.873447 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f118c89-c5d9-4939-b609-d3c173b27ae5-istiod-ca-cert\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-jxv68\" (UID: \"5f118c89-c5d9-4939-b609-d3c173b27ae5\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-jxv68" Nov 03 01:40:08.873514 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:08.873483 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f118c89-c5d9-4939-b609-d3c173b27ae5-workload-certs\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-jxv68\" (UID: \"5f118c89-c5d9-4939-b609-d3c173b27ae5\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-jxv68" Nov 03 01:40:08.873579 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:08.873556 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f118c89-c5d9-4939-b609-d3c173b27ae5-credential-socket\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-jxv68\" (UID: \"5f118c89-c5d9-4939-b609-d3c173b27ae5\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-jxv68" Nov 03 01:40:08.873630 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:08.873597 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2dsw\" (UniqueName: \"kubernetes.io/projected/5f118c89-c5d9-4939-b609-d3c173b27ae5-kube-api-access-h2dsw\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-jxv68\" (UID: \"5f118c89-c5d9-4939-b609-d3c173b27ae5\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-jxv68" Nov 03 01:40:08.873765 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:08.873743 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f118c89-c5d9-4939-b609-d3c173b27ae5-istio-token\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-jxv68\" (UID: \"5f118c89-c5d9-4939-b609-d3c173b27ae5\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-jxv68" Nov 03 01:40:08.873841 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:08.873789 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f118c89-c5d9-4939-b609-d3c173b27ae5-istio-data\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-jxv68\" (UID: \"5f118c89-c5d9-4939-b609-d3c173b27ae5\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-jxv68" Nov 03 01:40:08.873918 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:08.873904 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f118c89-c5d9-4939-b609-d3c173b27ae5-workload-socket\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-jxv68\" (UID: \"5f118c89-c5d9-4939-b609-d3c173b27ae5\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-jxv68" Nov 03 01:40:08.975580 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:08.975539 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f118c89-c5d9-4939-b609-d3c173b27ae5-istiod-ca-cert\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-jxv68\" (UID: \"5f118c89-c5d9-4939-b609-d3c173b27ae5\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-jxv68" Nov 03 01:40:08.975580 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:08.975586 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f118c89-c5d9-4939-b609-d3c173b27ae5-workload-certs\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-jxv68\" (UID: \"5f118c89-c5d9-4939-b609-d3c173b27ae5\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-jxv68" Nov 03 01:40:08.975852 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:08.975609 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f118c89-c5d9-4939-b609-d3c173b27ae5-credential-socket\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-jxv68\" (UID: \"5f118c89-c5d9-4939-b609-d3c173b27ae5\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-jxv68" Nov 03 01:40:08.975852 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:08.975747 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2dsw\" (UniqueName: \"kubernetes.io/projected/5f118c89-c5d9-4939-b609-d3c173b27ae5-kube-api-access-h2dsw\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-jxv68\" (UID: \"5f118c89-c5d9-4939-b609-d3c173b27ae5\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-jxv68" Nov 03 01:40:08.975973 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:08.975898 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f118c89-c5d9-4939-b609-d3c173b27ae5-istio-token\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-jxv68\" (UID: \"5f118c89-c5d9-4939-b609-d3c173b27ae5\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-jxv68" Nov 03 01:40:08.975973 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:08.975937 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f118c89-c5d9-4939-b609-d3c173b27ae5-istio-data\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-jxv68\" (UID: \"5f118c89-c5d9-4939-b609-d3c173b27ae5\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-jxv68" Nov 03 01:40:08.976081 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:08.976013 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f118c89-c5d9-4939-b609-d3c173b27ae5-workload-socket\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-jxv68\" (UID: \"5f118c89-c5d9-4939-b609-d3c173b27ae5\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-jxv68" Nov 03 01:40:08.976081 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:08.976067 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f118c89-c5d9-4939-b609-d3c173b27ae5-credential-socket\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-jxv68\" (UID: \"5f118c89-c5d9-4939-b609-d3c173b27ae5\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-jxv68" Nov 03 01:40:08.976188 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:08.976096 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f118c89-c5d9-4939-b609-d3c173b27ae5-workload-certs\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-jxv68\" (UID: \"5f118c89-c5d9-4939-b609-d3c173b27ae5\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-jxv68" Nov 03 01:40:08.976188 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:08.976123 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f118c89-c5d9-4939-b609-d3c173b27ae5-istio-envoy\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-jxv68\" (UID: \"5f118c89-c5d9-4939-b609-d3c173b27ae5\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-jxv68" Nov 03 01:40:08.976300 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:08.976200 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f118c89-c5d9-4939-b609-d3c173b27ae5-istio-podinfo\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-jxv68\" (UID: \"5f118c89-c5d9-4939-b609-d3c173b27ae5\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-jxv68" Nov 03 01:40:08.976399 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:08.976347 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f118c89-c5d9-4939-b609-d3c173b27ae5-istio-data\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-jxv68\" (UID: \"5f118c89-c5d9-4939-b609-d3c173b27ae5\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-jxv68" Nov 03 01:40:08.976484 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:08.976450 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f118c89-c5d9-4939-b609-d3c173b27ae5-istiod-ca-cert\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-jxv68\" (UID: \"5f118c89-c5d9-4939-b609-d3c173b27ae5\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-jxv68" Nov 03 01:40:08.976531 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:08.976473 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f118c89-c5d9-4939-b609-d3c173b27ae5-workload-socket\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-jxv68\" (UID: \"5f118c89-c5d9-4939-b609-d3c173b27ae5\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-jxv68" Nov 03 01:40:08.978520 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:08.978491 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f118c89-c5d9-4939-b609-d3c173b27ae5-istio-envoy\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-jxv68\" (UID: \"5f118c89-c5d9-4939-b609-d3c173b27ae5\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-jxv68" Nov 03 01:40:08.978719 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:08.978697 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f118c89-c5d9-4939-b609-d3c173b27ae5-istio-podinfo\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-jxv68\" (UID: \"5f118c89-c5d9-4939-b609-d3c173b27ae5\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-jxv68" Nov 03 01:40:09.038799 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:09.038754 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f118c89-c5d9-4939-b609-d3c173b27ae5-istio-token\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-jxv68\" (UID: \"5f118c89-c5d9-4939-b609-d3c173b27ae5\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-jxv68" Nov 03 01:40:09.038963 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:09.038897 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2dsw\" (UniqueName: \"kubernetes.io/projected/5f118c89-c5d9-4939-b609-d3c173b27ae5-kube-api-access-h2dsw\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-jxv68\" (UID: \"5f118c89-c5d9-4939-b609-d3c173b27ae5\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-jxv68" Nov 03 01:40:09.066298 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:09.066199 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:09.125708 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:09.125668 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-jxv68" Nov 03 01:40:09.562912 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:40:09.562867 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f118c89_c5d9_4939_b609_d3c173b27ae5.slice/crio-7452b83aa99dc4b968424e5bfda01396941af2ea71b048aff10757bf5fe809e0 WatchSource:0}: Error finding container 7452b83aa99dc4b968424e5bfda01396941af2ea71b048aff10757bf5fe809e0: Status 404 returned error can't find the container with id 7452b83aa99dc4b968424e5bfda01396941af2ea71b048aff10757bf5fe809e0 Nov 03 01:40:09.565553 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:09.565514 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 01:40:09.565669 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:09.565600 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 01:40:09.565669 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:09.565646 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 01:40:09.566046 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:09.566020 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-jxv68"] Nov 03 01:40:09.785052 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:09.785012 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:09.869227 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:09.869134 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-jxv68" event={"ID":"5f118c89-c5d9-4939-b609-d3c173b27ae5","Type":"ContainerStarted","Data":"e37358d2a8a5f7ced043fd9922e3dc4634021ea0e09c7532cd0dd910f4054f0e"} Nov 03 01:40:09.869227 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:09.869199 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-jxv68" event={"ID":"5f118c89-c5d9-4939-b609-d3c173b27ae5","Type":"ContainerStarted","Data":"7452b83aa99dc4b968424e5bfda01396941af2ea71b048aff10757bf5fe809e0"} Nov 03 01:40:09.994715 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:09.994657 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-jxv68" podStartSLOduration=1.994640727 podStartE2EDuration="1.994640727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:40:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:40:09.992184101 +0000 UTC m=+1292.021666416" watchObservedRunningTime="2026-11-03 01:40:09.994640727 +0000 UTC m=+1292.024123022" Nov 03 01:40:10.005559 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:10.005517 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:10.126450 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:10.126318 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-jxv68" Nov 03 01:40:10.127976 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:10.127948 5437 patch_prober.go:29] interesting pod/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-jxv68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 03 01:40:10.128090 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:10.127998 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-jxv68" podUID="5f118c89-c5d9-4939-b609-d3c173b27ae5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 03 01:40:10.413192 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:10.413088 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:10.642080 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:10.642037 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:10.854096 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:10.854063 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:11.063999 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:11.063964 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:11.145980 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:11.145840 5437 patch_prober.go:29] interesting pod/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-jxv68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 03 01:40:11.146395 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:11.146345 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-jxv68" podUID="5f118c89-c5d9-4939-b609-d3c173b27ae5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 03 01:40:12.127888 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:12.127847 5437 patch_prober.go:29] interesting pod/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-jxv68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 03 01:40:12.128616 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:12.128577 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-jxv68" podUID="5f118c89-c5d9-4939-b609-d3c173b27ae5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 03 01:40:13.136392 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:13.134827 5437 patch_prober.go:29] interesting pod/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-jxv68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 03 01:40:13.136392 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:13.134907 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-jxv68" podUID="5f118c89-c5d9-4939-b609-d3c173b27ae5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 03 01:40:14.063816 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:40:14.063750 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 01:40:14.143440 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:14.143397 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:14.144792 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:14.144328 5437 patch_prober.go:29] interesting pod/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-jxv68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 03 01:40:14.144792 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:14.144406 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-jxv68" podUID="5f118c89-c5d9-4939-b609-d3c173b27ae5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 03 01:40:15.133841 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:15.133798 5437 patch_prober.go:29] interesting pod/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-jxv68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 03 01:40:15.134979 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:15.134943 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-jxv68" podUID="5f118c89-c5d9-4939-b609-d3c173b27ae5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 03 01:40:16.126675 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:16.126632 5437 patch_prober.go:29] interesting pod/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-jxv68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 03 01:40:16.129232 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:16.126692 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-jxv68" podUID="5f118c89-c5d9-4939-b609-d3c173b27ae5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 03 01:40:16.506411 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:16.506286 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:16.736220 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:16.736167 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:17.235031 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:17.234997 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-jxv68" Nov 03 01:40:17.235686 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:17.235664 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-jxv68" Nov 03 01:40:17.236620 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:17.236598 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-jxv68" Nov 03 01:40:19.319878 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:19.319824 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:19.668918 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:19.668833 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:19.875917 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:19.875880 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:20.021476 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:20.021342 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2b9f22c1-7cd5c774dc-xbtwr"] Nov 03 01:40:20.026430 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:20.026398 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-2b9f22c1-7cd5c774dc-xbtwr" Nov 03 01:40:20.082155 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:20.082115 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2b9f22c1-7cd5c774dc-xbtwr"] Nov 03 01:40:20.129161 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:20.129121 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-2b9f22c1\"" Nov 03 01:40:20.130411 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:20.130386 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-2b9f22c1-dockercfg-xnk5r\"" Nov 03 01:40:20.147937 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:20.147899 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18277241-9019-4b65-86b7-7084af0b742d-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-2b9f22c1-7cd5c774dc-xbtwr\" (UID: \"18277241-9019-4b65-86b7-7084af0b742d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2b9f22c1-7cd5c774dc-xbtwr" Nov 03 01:40:20.148113 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:20.148078 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwrks\" (UniqueName: \"kubernetes.io/projected/18277241-9019-4b65-86b7-7084af0b742d-kube-api-access-nwrks\") pod \"payload-pre-processing-e2e-mt-2b9f22c1-7cd5c774dc-xbtwr\" (UID: \"18277241-9019-4b65-86b7-7084af0b742d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2b9f22c1-7cd5c774dc-xbtwr" Nov 03 01:40:20.251642 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:20.249855 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18277241-9019-4b65-86b7-7084af0b742d-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-2b9f22c1-7cd5c774dc-xbtwr\" (UID: \"18277241-9019-4b65-86b7-7084af0b742d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2b9f22c1-7cd5c774dc-xbtwr" Nov 03 01:40:20.251642 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:20.249983 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwrks\" (UniqueName: \"kubernetes.io/projected/18277241-9019-4b65-86b7-7084af0b742d-kube-api-access-nwrks\") pod \"payload-pre-processing-e2e-mt-2b9f22c1-7cd5c774dc-xbtwr\" (UID: \"18277241-9019-4b65-86b7-7084af0b742d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2b9f22c1-7cd5c774dc-xbtwr" Nov 03 01:40:20.251642 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:20.251380 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18277241-9019-4b65-86b7-7084af0b742d-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-2b9f22c1-7cd5c774dc-xbtwr\" (UID: \"18277241-9019-4b65-86b7-7084af0b742d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2b9f22c1-7cd5c774dc-xbtwr" Nov 03 01:40:20.313826 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:20.313788 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwrks\" (UniqueName: \"kubernetes.io/projected/18277241-9019-4b65-86b7-7084af0b742d-kube-api-access-nwrks\") pod \"payload-pre-processing-e2e-mt-2b9f22c1-7cd5c774dc-xbtwr\" (UID: \"18277241-9019-4b65-86b7-7084af0b742d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2b9f22c1-7cd5c774dc-xbtwr" Nov 03 01:40:20.348473 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:20.348428 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-2b9f22c1-7cd5c774dc-xbtwr" Nov 03 01:40:20.739336 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:40:20.739292 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18277241_9019_4b65_86b7_7084af0b742d.slice/crio-eea6add51c3925f0e21b7a67e96a187d9fbb283bb82a312fddd42c4ef735ad95 WatchSource:0}: Error finding container eea6add51c3925f0e21b7a67e96a187d9fbb283bb82a312fddd42c4ef735ad95: Status 404 returned error can't find the container with id eea6add51c3925f0e21b7a67e96a187d9fbb283bb82a312fddd42c4ef735ad95 Nov 03 01:40:20.746155 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:20.746123 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2b9f22c1-7cd5c774dc-xbtwr"] Nov 03 01:40:20.945667 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:20.945625 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-2b9f22c1-7cd5c774dc-xbtwr" event={"ID":"18277241-9019-4b65-86b7-7084af0b742d","Type":"ContainerStarted","Data":"ea4323a5a1d4f1e24a8d1e202b285fb2f3b47673dbb910e48df50b1e9ec430cd"} Nov 03 01:40:20.945868 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:20.945684 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-2b9f22c1-7cd5c774dc-xbtwr" event={"ID":"18277241-9019-4b65-86b7-7084af0b742d","Type":"ContainerStarted","Data":"eea6add51c3925f0e21b7a67e96a187d9fbb283bb82a312fddd42c4ef735ad95"} Nov 03 01:40:20.945868 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:20.945859 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-2b9f22c1-7cd5c774dc-xbtwr" Nov 03 01:40:21.066848 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:21.066793 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-2b9f22c1-7cd5c774dc-xbtwr" podStartSLOduration=2.066776534 podStartE2EDuration="2.066776534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:40:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:40:21.064738546 +0000 UTC m=+1303.094220840" watchObservedRunningTime="2026-11-03 01:40:21.066776534 +0000 UTC m=+1303.096258830" Nov 03 01:40:22.510473 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:22.510435 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-e1793d/distinct-2-w1-e1793d-kserve-84fcdfb9fd-r27tq"] Nov 03 01:40:22.514797 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:22.514774 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e1793d/distinct-2-w1-e1793d-kserve-84fcdfb9fd-r27tq" Nov 03 01:40:22.571175 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:22.571141 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-e1793d/distinct-2-w1-e1793d-kserve-84fcdfb9fd-r27tq"] Nov 03 01:40:22.582834 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:22.582797 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2cf938f2-75d3-4af7-8645-cdf69a1d2744-home\") pod \"distinct-2-w1-e1793d-kserve-84fcdfb9fd-r27tq\" (UID: \"2cf938f2-75d3-4af7-8645-cdf69a1d2744\") " pod="e2e-models-e2e-worker-w1-e1793d/distinct-2-w1-e1793d-kserve-84fcdfb9fd-r27tq" Nov 03 01:40:22.583036 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:22.582958 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2cf938f2-75d3-4af7-8645-cdf69a1d2744-tls-certs\") pod \"distinct-2-w1-e1793d-kserve-84fcdfb9fd-r27tq\" (UID: \"2cf938f2-75d3-4af7-8645-cdf69a1d2744\") " pod="e2e-models-e2e-worker-w1-e1793d/distinct-2-w1-e1793d-kserve-84fcdfb9fd-r27tq" Nov 03 01:40:22.583155 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:22.583130 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2cf938f2-75d3-4af7-8645-cdf69a1d2744-dshm\") pod \"distinct-2-w1-e1793d-kserve-84fcdfb9fd-r27tq\" (UID: \"2cf938f2-75d3-4af7-8645-cdf69a1d2744\") " pod="e2e-models-e2e-worker-w1-e1793d/distinct-2-w1-e1793d-kserve-84fcdfb9fd-r27tq" Nov 03 01:40:22.583271 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:22.583252 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cf938f2-75d3-4af7-8645-cdf69a1d2744-tmp-dir\") pod \"distinct-2-w1-e1793d-kserve-84fcdfb9fd-r27tq\" (UID: \"2cf938f2-75d3-4af7-8645-cdf69a1d2744\") " pod="e2e-models-e2e-worker-w1-e1793d/distinct-2-w1-e1793d-kserve-84fcdfb9fd-r27tq" Nov 03 01:40:22.583338 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:22.583286 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cf938f2-75d3-4af7-8645-cdf69a1d2744-model-cache\") pod \"distinct-2-w1-e1793d-kserve-84fcdfb9fd-r27tq\" (UID: \"2cf938f2-75d3-4af7-8645-cdf69a1d2744\") " pod="e2e-models-e2e-worker-w1-e1793d/distinct-2-w1-e1793d-kserve-84fcdfb9fd-r27tq" Nov 03 01:40:22.583422 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:22.583349 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6v8rs\" (UniqueName: \"kubernetes.io/projected/2cf938f2-75d3-4af7-8645-cdf69a1d2744-kube-api-access-6v8rs\") pod \"distinct-2-w1-e1793d-kserve-84fcdfb9fd-r27tq\" (UID: \"2cf938f2-75d3-4af7-8645-cdf69a1d2744\") " pod="e2e-models-e2e-worker-w1-e1793d/distinct-2-w1-e1793d-kserve-84fcdfb9fd-r27tq" Nov 03 01:40:22.619258 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:22.619226 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-e1793d\"/\"distinct-2-w1-e1793d-kserve-self-signed-certs\"" Nov 03 01:40:22.666327 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:22.666278 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:22.684900 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:22.684859 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2cf938f2-75d3-4af7-8645-cdf69a1d2744-tls-certs\") pod \"distinct-2-w1-e1793d-kserve-84fcdfb9fd-r27tq\" (UID: \"2cf938f2-75d3-4af7-8645-cdf69a1d2744\") " pod="e2e-models-e2e-worker-w1-e1793d/distinct-2-w1-e1793d-kserve-84fcdfb9fd-r27tq" Nov 03 01:40:22.685104 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:22.684951 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2cf938f2-75d3-4af7-8645-cdf69a1d2744-dshm\") pod \"distinct-2-w1-e1793d-kserve-84fcdfb9fd-r27tq\" (UID: \"2cf938f2-75d3-4af7-8645-cdf69a1d2744\") " pod="e2e-models-e2e-worker-w1-e1793d/distinct-2-w1-e1793d-kserve-84fcdfb9fd-r27tq" Nov 03 01:40:22.685104 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:22.685037 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cf938f2-75d3-4af7-8645-cdf69a1d2744-tmp-dir\") pod \"distinct-2-w1-e1793d-kserve-84fcdfb9fd-r27tq\" (UID: \"2cf938f2-75d3-4af7-8645-cdf69a1d2744\") " pod="e2e-models-e2e-worker-w1-e1793d/distinct-2-w1-e1793d-kserve-84fcdfb9fd-r27tq" Nov 03 01:40:22.685104 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:22.685066 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cf938f2-75d3-4af7-8645-cdf69a1d2744-model-cache\") pod \"distinct-2-w1-e1793d-kserve-84fcdfb9fd-r27tq\" (UID: \"2cf938f2-75d3-4af7-8645-cdf69a1d2744\") " pod="e2e-models-e2e-worker-w1-e1793d/distinct-2-w1-e1793d-kserve-84fcdfb9fd-r27tq" Nov 03 01:40:22.685265 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:22.685107 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6v8rs\" (UniqueName: \"kubernetes.io/projected/2cf938f2-75d3-4af7-8645-cdf69a1d2744-kube-api-access-6v8rs\") pod \"distinct-2-w1-e1793d-kserve-84fcdfb9fd-r27tq\" (UID: \"2cf938f2-75d3-4af7-8645-cdf69a1d2744\") " pod="e2e-models-e2e-worker-w1-e1793d/distinct-2-w1-e1793d-kserve-84fcdfb9fd-r27tq" Nov 03 01:40:22.685265 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:22.685166 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2cf938f2-75d3-4af7-8645-cdf69a1d2744-home\") pod \"distinct-2-w1-e1793d-kserve-84fcdfb9fd-r27tq\" (UID: \"2cf938f2-75d3-4af7-8645-cdf69a1d2744\") " pod="e2e-models-e2e-worker-w1-e1793d/distinct-2-w1-e1793d-kserve-84fcdfb9fd-r27tq" Nov 03 01:40:22.685578 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:22.685548 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cf938f2-75d3-4af7-8645-cdf69a1d2744-tmp-dir\") pod \"distinct-2-w1-e1793d-kserve-84fcdfb9fd-r27tq\" (UID: \"2cf938f2-75d3-4af7-8645-cdf69a1d2744\") " pod="e2e-models-e2e-worker-w1-e1793d/distinct-2-w1-e1793d-kserve-84fcdfb9fd-r27tq" Nov 03 01:40:22.685681 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:22.685614 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cf938f2-75d3-4af7-8645-cdf69a1d2744-model-cache\") pod \"distinct-2-w1-e1793d-kserve-84fcdfb9fd-r27tq\" (UID: \"2cf938f2-75d3-4af7-8645-cdf69a1d2744\") " pod="e2e-models-e2e-worker-w1-e1793d/distinct-2-w1-e1793d-kserve-84fcdfb9fd-r27tq" Nov 03 01:40:22.685681 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:22.685632 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2cf938f2-75d3-4af7-8645-cdf69a1d2744-home\") pod \"distinct-2-w1-e1793d-kserve-84fcdfb9fd-r27tq\" (UID: \"2cf938f2-75d3-4af7-8645-cdf69a1d2744\") " pod="e2e-models-e2e-worker-w1-e1793d/distinct-2-w1-e1793d-kserve-84fcdfb9fd-r27tq" Nov 03 01:40:22.687563 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:22.687532 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2cf938f2-75d3-4af7-8645-cdf69a1d2744-dshm\") pod \"distinct-2-w1-e1793d-kserve-84fcdfb9fd-r27tq\" (UID: \"2cf938f2-75d3-4af7-8645-cdf69a1d2744\") " pod="e2e-models-e2e-worker-w1-e1793d/distinct-2-w1-e1793d-kserve-84fcdfb9fd-r27tq" Nov 03 01:40:22.688166 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:22.688134 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2cf938f2-75d3-4af7-8645-cdf69a1d2744-tls-certs\") pod \"distinct-2-w1-e1793d-kserve-84fcdfb9fd-r27tq\" (UID: \"2cf938f2-75d3-4af7-8645-cdf69a1d2744\") " pod="e2e-models-e2e-worker-w1-e1793d/distinct-2-w1-e1793d-kserve-84fcdfb9fd-r27tq" Nov 03 01:40:22.749580 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:22.749526 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6v8rs\" (UniqueName: \"kubernetes.io/projected/2cf938f2-75d3-4af7-8645-cdf69a1d2744-kube-api-access-6v8rs\") pod \"distinct-2-w1-e1793d-kserve-84fcdfb9fd-r27tq\" (UID: \"2cf938f2-75d3-4af7-8645-cdf69a1d2744\") " pod="e2e-models-e2e-worker-w1-e1793d/distinct-2-w1-e1793d-kserve-84fcdfb9fd-r27tq" Nov 03 01:40:22.827478 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:22.827440 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e1793d/distinct-2-w1-e1793d-kserve-84fcdfb9fd-r27tq" Nov 03 01:40:23.202678 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:40:23.202646 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2cf938f2_75d3_4af7_8645_cdf69a1d2744.slice/crio-a280bfe8b3a2c47a92dcccf12e40cba3571d53ca688f505136fcd50fc81a250f WatchSource:0}: Error finding container a280bfe8b3a2c47a92dcccf12e40cba3571d53ca688f505136fcd50fc81a250f: Status 404 returned error can't find the container with id a280bfe8b3a2c47a92dcccf12e40cba3571d53ca688f505136fcd50fc81a250f Nov 03 01:40:23.206276 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:23.206252 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-e1793d/distinct-2-w1-e1793d-kserve-84fcdfb9fd-r27tq"] Nov 03 01:40:23.972440 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:23.972388 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e1793d/distinct-2-w1-e1793d-kserve-84fcdfb9fd-r27tq" event={"ID":"2cf938f2-75d3-4af7-8645-cdf69a1d2744","Type":"ContainerStarted","Data":"01d98296a853cda31ecc51480f7034e42dbd9e85f5a20127e8aeb160782efeb9"} Nov 03 01:40:23.972440 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:23.972444 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e1793d/distinct-2-w1-e1793d-kserve-84fcdfb9fd-r27tq" event={"ID":"2cf938f2-75d3-4af7-8645-cdf69a1d2744","Type":"ContainerStarted","Data":"a280bfe8b3a2c47a92dcccf12e40cba3571d53ca688f505136fcd50fc81a250f"} Nov 03 01:40:24.096695 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:24.096639 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-e1793d/distinct-2-w1-e1793d-kserve-84fcdfb9fd-r27tq" podStartSLOduration=2.096625777 podStartE2EDuration="2.096625777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:40:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:40:24.095642505 +0000 UTC m=+1306.125124801" watchObservedRunningTime="2026-11-03 01:40:24.096625777 +0000 UTC m=+1306.126108099" Nov 03 01:40:24.128295 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:24.128259 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:24.334251 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:24.334208 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:24.543803 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:24.543765 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:24.827751 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:24.827706 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e1793d/distinct-2-w1-e1793d-kserve-84fcdfb9fd-r27tq" Nov 03 01:40:28.464391 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:28.463423 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:29.005390 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:29.004463 5437 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-41aa5f/simulator-w3-41aa5f-kserve-586cfbcdd4-gsmfk" podUID="8a87f809-0b26-480e-bb37-ff234a40c5c5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:40:29.077538 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:29.077483 5437 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l87z7" podUID="03a82c52-52db-44b7-bf71-64ec0c963054" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:40:29.133554 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:29.133501 5437 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e1793d/simulator-w1-e1793d-kserve-7595dfc6b7-dbbgf" podUID="f28ccf15-b92d-400c-9204-69b66279d7d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 01:40:29.267594 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:29.267471 5437 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-41aa5f/premium-w3-41aa5f-kserve-5fb94fd5fc-gmtkb" podUID="c46dca02-6036-46de-b0f3-df796d231f62" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:40:29.283513 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:29.283462 5437 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0bf34a/simulator-w4-0bf34a-kserve-6bc96559d9-gkdwl" podUID="4cfca4a1-0da6-4637-bc48-8e65776b9ca9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Nov 03 01:40:29.354517 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:29.354461 5437 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e1793d/distinct-w1-e1793d-kserve-687855c49-vn8zs" podUID="a821ee29-70aa-4cad-87d4-3fb0edd20810" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:40:30.196390 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:30.096947 5437 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l87z7" podUID="03a82c52-52db-44b7-bf71-64ec0c963054" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": context deadline exceeded" Nov 03 01:40:30.227672 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:30.188652 5437 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e1793d/simulator-w1-e1793d-kserve-7595dfc6b7-dbbgf" podUID="f28ccf15-b92d-400c-9204-69b66279d7d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Nov 03 01:40:30.270094 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:30.270039 5437 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-41aa5f/premium-w3-41aa5f-kserve-5fb94fd5fc-gmtkb" podUID="c46dca02-6036-46de-b0f3-df796d231f62" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:40:30.288502 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:30.288453 5437 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0bf34a/simulator-w4-0bf34a-kserve-6bc96559d9-gkdwl" podUID="4cfca4a1-0da6-4637-bc48-8e65776b9ca9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:40:30.356757 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:30.356699 5437 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e1793d/distinct-w1-e1793d-kserve-687855c49-vn8zs" podUID="a821ee29-70aa-4cad-87d4-3fb0edd20810" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:40:30.428387 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:30.421462 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e1793d/simulator-w1-e1793d-kserve-7595dfc6b7-dbbgf" Nov 03 01:40:30.428387 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:30.421529 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l87z7" Nov 03 01:40:30.428387 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:30.421565 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-41aa5f/premium-w3-41aa5f-kserve-5fb94fd5fc-gmtkb" Nov 03 01:40:30.428387 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:30.421595 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0bf34a/simulator-w4-0bf34a-kserve-6bc96559d9-gkdwl" Nov 03 01:40:30.428387 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:30.421627 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e1793d/distinct-w1-e1793d-kserve-687855c49-vn8zs" Nov 03 01:40:30.428387 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:30.421659 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-41aa5f/premium-w3-41aa5f-kserve-5fb94fd5fc-gmtkb" Nov 03 01:40:30.428387 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:30.421691 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e1793d/distinct-w1-e1793d-kserve-687855c49-vn8zs" Nov 03 01:40:30.438036 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:30.437964 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:30.441837 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:30.441810 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0bf34a/simulator-w4-0bf34a-kserve-6bc96559d9-gkdwl" Nov 03 01:40:30.738973 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:30.738937 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l87z7" Nov 03 01:40:31.033988 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:31.033889 5437 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-41aa5f/simulator-w3-41aa5f-kserve-586cfbcdd4-gsmfk" podUID="8a87f809-0b26-480e-bb37-ff234a40c5c5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:40:31.078387 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:31.069204 5437 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-41aa5f/simulator-w3-41aa5f-kserve-586cfbcdd4-gsmfk" podUID="8a87f809-0b26-480e-bb37-ff234a40c5c5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:40:31.114445 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:31.114418 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e1793d/simulator-w1-e1793d-kserve-7595dfc6b7-dbbgf" Nov 03 01:40:31.954993 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:31.954965 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-2b9f22c1-7cd5c774dc-xbtwr" Nov 03 01:40:32.828515 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:32.828464 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-e1793d/distinct-2-w1-e1793d-kserve-84fcdfb9fd-r27tq" Nov 03 01:40:32.844406 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:32.844355 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-e1793d/distinct-2-w1-e1793d-kserve-84fcdfb9fd-r27tq" Nov 03 01:40:33.424906 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:33.424877 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e1793d/distinct-2-w1-e1793d-kserve-84fcdfb9fd-r27tq" Nov 03 01:40:34.748641 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:34.748605 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:35.075072 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:35.075042 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:35.483378 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:35.483278 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:35.683030 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:35.682989 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:40.200994 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:40.200953 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:43.092200 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:43.092164 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:44.036548 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:40:44.036505 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 01:40:44.177737 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:44.177705 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-d8gw2"] Nov 03 01:40:44.182288 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:44.182266 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-d8gw2" Nov 03 01:40:44.236650 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:44.236615 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 03 01:40:44.236650 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:44.236615 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 03 01:40:44.236861 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:44.236657 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-h4j6w\"" Nov 03 01:40:44.236861 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:44.236714 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 03 01:40:44.283003 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:44.282968 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5ed0a52-1e65-46f3-b268-bdb523bfec88-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-d8gw2\" (UID: \"c5ed0a52-1e65-46f3-b268-bdb523bfec88\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-d8gw2" Nov 03 01:40:44.283192 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:44.283033 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpb6v\" (UniqueName: \"kubernetes.io/projected/c5ed0a52-1e65-46f3-b268-bdb523bfec88-kube-api-access-gpb6v\") pod \"istiod-openshift-gateway-7ddd9cfdc8-d8gw2\" (UID: \"c5ed0a52-1e65-46f3-b268-bdb523bfec88\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-d8gw2" Nov 03 01:40:44.283192 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:44.283136 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c5ed0a52-1e65-46f3-b268-bdb523bfec88-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-d8gw2\" (UID: \"c5ed0a52-1e65-46f3-b268-bdb523bfec88\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-d8gw2" Nov 03 01:40:44.283192 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:44.283170 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5ed0a52-1e65-46f3-b268-bdb523bfec88-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-d8gw2\" (UID: \"c5ed0a52-1e65-46f3-b268-bdb523bfec88\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-d8gw2" Nov 03 01:40:44.283414 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:44.283242 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c5ed0a52-1e65-46f3-b268-bdb523bfec88-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-d8gw2\" (UID: \"c5ed0a52-1e65-46f3-b268-bdb523bfec88\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-d8gw2" Nov 03 01:40:44.283414 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:44.283273 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c5ed0a52-1e65-46f3-b268-bdb523bfec88-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-d8gw2\" (UID: \"c5ed0a52-1e65-46f3-b268-bdb523bfec88\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-d8gw2" Nov 03 01:40:44.283414 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:44.283355 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c5ed0a52-1e65-46f3-b268-bdb523bfec88-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-d8gw2\" (UID: \"c5ed0a52-1e65-46f3-b268-bdb523bfec88\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-d8gw2" Nov 03 01:40:44.291700 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:44.291623 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-d8gw2"] Nov 03 01:40:44.384999 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:44.384960 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c5ed0a52-1e65-46f3-b268-bdb523bfec88-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-d8gw2\" (UID: \"c5ed0a52-1e65-46f3-b268-bdb523bfec88\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-d8gw2" Nov 03 01:40:44.385183 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:44.385072 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5ed0a52-1e65-46f3-b268-bdb523bfec88-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-d8gw2\" (UID: \"c5ed0a52-1e65-46f3-b268-bdb523bfec88\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-d8gw2" Nov 03 01:40:44.385183 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:44.385113 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpb6v\" (UniqueName: \"kubernetes.io/projected/c5ed0a52-1e65-46f3-b268-bdb523bfec88-kube-api-access-gpb6v\") pod \"istiod-openshift-gateway-7ddd9cfdc8-d8gw2\" (UID: \"c5ed0a52-1e65-46f3-b268-bdb523bfec88\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-d8gw2" Nov 03 01:40:44.385263 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:44.385247 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c5ed0a52-1e65-46f3-b268-bdb523bfec88-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-d8gw2\" (UID: \"c5ed0a52-1e65-46f3-b268-bdb523bfec88\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-d8gw2" Nov 03 01:40:44.385299 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:44.385286 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5ed0a52-1e65-46f3-b268-bdb523bfec88-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-d8gw2\" (UID: \"c5ed0a52-1e65-46f3-b268-bdb523bfec88\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-d8gw2" Nov 03 01:40:44.385356 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:44.385342 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c5ed0a52-1e65-46f3-b268-bdb523bfec88-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-d8gw2\" (UID: \"c5ed0a52-1e65-46f3-b268-bdb523bfec88\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-d8gw2" Nov 03 01:40:44.385507 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:44.385478 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c5ed0a52-1e65-46f3-b268-bdb523bfec88-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-d8gw2\" (UID: \"c5ed0a52-1e65-46f3-b268-bdb523bfec88\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-d8gw2" Nov 03 01:40:44.386382 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:44.386332 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c5ed0a52-1e65-46f3-b268-bdb523bfec88-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-d8gw2\" (UID: \"c5ed0a52-1e65-46f3-b268-bdb523bfec88\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-d8gw2" Nov 03 01:40:44.387632 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:44.387608 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5ed0a52-1e65-46f3-b268-bdb523bfec88-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-d8gw2\" (UID: \"c5ed0a52-1e65-46f3-b268-bdb523bfec88\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-d8gw2" Nov 03 01:40:44.387842 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:44.387821 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c5ed0a52-1e65-46f3-b268-bdb523bfec88-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-d8gw2\" (UID: \"c5ed0a52-1e65-46f3-b268-bdb523bfec88\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-d8gw2" Nov 03 01:40:44.387894 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:44.387826 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c5ed0a52-1e65-46f3-b268-bdb523bfec88-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-d8gw2\" (UID: \"c5ed0a52-1e65-46f3-b268-bdb523bfec88\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-d8gw2" Nov 03 01:40:44.387936 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:44.387885 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c5ed0a52-1e65-46f3-b268-bdb523bfec88-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-d8gw2\" (UID: \"c5ed0a52-1e65-46f3-b268-bdb523bfec88\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-d8gw2" Nov 03 01:40:44.448329 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:44.448296 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5ed0a52-1e65-46f3-b268-bdb523bfec88-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-d8gw2\" (UID: \"c5ed0a52-1e65-46f3-b268-bdb523bfec88\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-d8gw2" Nov 03 01:40:44.448525 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:44.448500 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpb6v\" (UniqueName: \"kubernetes.io/projected/c5ed0a52-1e65-46f3-b268-bdb523bfec88-kube-api-access-gpb6v\") pod \"istiod-openshift-gateway-7ddd9cfdc8-d8gw2\" (UID: \"c5ed0a52-1e65-46f3-b268-bdb523bfec88\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-d8gw2" Nov 03 01:40:44.495197 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:44.495158 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-d8gw2" Nov 03 01:40:44.917178 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:40:44.917096 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5ed0a52_1e65_46f3_b268_bdb523bfec88.slice/crio-bac2f5feb0a32247ed25ffb61b43231dc9243d959ee4782a7ab4889104244e2c WatchSource:0}: Error finding container bac2f5feb0a32247ed25ffb61b43231dc9243d959ee4782a7ab4889104244e2c: Status 404 returned error can't find the container with id bac2f5feb0a32247ed25ffb61b43231dc9243d959ee4782a7ab4889104244e2c Nov 03 01:40:44.920077 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:44.920042 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 01:40:44.920178 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:44.920112 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 01:40:44.920945 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:44.920920 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-d8gw2"] Nov 03 01:40:44.987735 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:44.987706 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:45.500421 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:45.500344 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-d8gw2" event={"ID":"c5ed0a52-1e65-46f3-b268-bdb523bfec88","Type":"ContainerStarted","Data":"1987b276e5b3148902e698486c70de9265e2b2e6bb9c3b9c85a7f8f2337da888"} Nov 03 01:40:45.500918 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:45.500890 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-d8gw2" Nov 03 01:40:45.501030 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:45.500922 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-d8gw2" event={"ID":"c5ed0a52-1e65-46f3-b268-bdb523bfec88","Type":"ContainerStarted","Data":"bac2f5feb0a32247ed25ffb61b43231dc9243d959ee4782a7ab4889104244e2c"} Nov 03 01:40:45.502799 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:45.502759 5437 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-d8gw2 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 01:40:45.502926 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:45.502830 5437 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-d8gw2" podUID="c5ed0a52-1e65-46f3-b268-bdb523bfec88" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 01:40:45.628053 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:45.627993 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-d8gw2" podStartSLOduration=1.627976788 podStartE2EDuration="1.627976788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:40:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:40:45.624636026 +0000 UTC m=+1327.654118319" watchObservedRunningTime="2026-11-03 01:40:45.627976788 +0000 UTC m=+1327.657459080" Nov 03 01:40:46.510332 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:46.510284 5437 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-d8gw2 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 01:40:46.510871 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:46.510346 5437 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-d8gw2" podUID="c5ed0a52-1e65-46f3-b268-bdb523bfec88" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 01:40:49.510644 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:49.510608 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-d8gw2" Nov 03 01:40:53.594453 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:53.594412 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:53.802130 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:53.802092 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:54.706036 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:54.705985 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:56.785564 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:56.785525 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:59.671082 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:40:59.671043 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:02.694836 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:41:02.694781 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:04.475726 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:41:04.475690 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:09.287171 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:41:09.287119 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:14.036747 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:41:14.036706 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 01:41:14.105843 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:41:14.105772 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:18.891076 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:41:18.890994 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:18.989261 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:41:18.989222 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mdjkm"] Nov 03 01:41:18.995112 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:41:18.995084 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mdjkm" Nov 03 01:41:19.040892 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:41:19.040846 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xb7x2"] Nov 03 01:41:19.045135 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:41:19.045111 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xb7x2" Nov 03 01:41:19.096145 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:41:19.096100 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mdjkm"] Nov 03 01:41:19.102494 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:41:19.102454 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be9a9975-2bf4-47b3-918f-6ef9e4a88bca-istio-envoy\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mdjkm\" (UID: \"be9a9975-2bf4-47b3-918f-6ef9e4a88bca\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mdjkm" Nov 03 01:41:19.102494 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:41:19.102495 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be9a9975-2bf4-47b3-918f-6ef9e4a88bca-istio-data\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mdjkm\" (UID: \"be9a9975-2bf4-47b3-918f-6ef9e4a88bca\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mdjkm" Nov 03 01:41:19.102775 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:41:19.102618 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be9a9975-2bf4-47b3-918f-6ef9e4a88bca-istio-token\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mdjkm\" (UID: \"be9a9975-2bf4-47b3-918f-6ef9e4a88bca\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mdjkm" Nov 03 01:41:19.102775 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:41:19.102652 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be9a9975-2bf4-47b3-918f-6ef9e4a88bca-istiod-ca-cert\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mdjkm\" (UID: \"be9a9975-2bf4-47b3-918f-6ef9e4a88bca\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mdjkm" Nov 03 01:41:19.102775 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:41:19.102728 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-464nr\" (UniqueName: \"kubernetes.io/projected/be9a9975-2bf4-47b3-918f-6ef9e4a88bca-kube-api-access-464nr\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mdjkm\" (UID: \"be9a9975-2bf4-47b3-918f-6ef9e4a88bca\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mdjkm" Nov 03 01:41:19.102939 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:41:19.102863 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be9a9975-2bf4-47b3-918f-6ef9e4a88bca-credential-socket\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mdjkm\" (UID: \"be9a9975-2bf4-47b3-918f-6ef9e4a88bca\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mdjkm" Nov 03 01:41:19.102939 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:41:19.102897 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be9a9975-2bf4-47b3-918f-6ef9e4a88bca-workload-socket\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mdjkm\" (UID: \"be9a9975-2bf4-47b3-918f-6ef9e4a88bca\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mdjkm" Nov 03 01:41:19.103049 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:41:19.103000 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be9a9975-2bf4-47b3-918f-6ef9e4a88bca-istio-podinfo\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mdjkm\" (UID: \"be9a9975-2bf4-47b3-918f-6ef9e4a88bca\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mdjkm" Nov 03 01:41:19.103049 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:41:19.103021 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be9a9975-2bf4-47b3-918f-6ef9e4a88bca-workload-certs\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mdjkm\" (UID: \"be9a9975-2bf4-47b3-918f-6ef9e4a88bca\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mdjkm" Nov 03 01:41:19.140354 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:41:19.140314 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xb7x2"] Nov 03 01:41:19.204527 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:41:19.204424 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be9a9975-2bf4-47b3-918f-6ef9e4a88bca-credential-socket\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mdjkm\" (UID: \"be9a9975-2bf4-47b3-918f-6ef9e4a88bca\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mdjkm" Nov 03 01:41:19.204527 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:41:19.204469 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b4a66c8-89c8-4894-9244-2a0c3587a06c-istio-envoy\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xb7x2\" (UID: \"4b4a66c8-89c8-4894-9244-2a0c3587a06c\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xb7x2" Nov 03 01:41:19.204527 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:41:19.204506 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be9a9975-2bf4-47b3-918f-6ef9e4a88bca-workload-socket\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mdjkm\" (UID: \"be9a9975-2bf4-47b3-918f-6ef9e4a88bca\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mdjkm" Nov 03 01:41:19.204798 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:41:19.204659 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be9a9975-2bf4-47b3-918f-6ef9e4a88bca-istio-podinfo\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mdjkm\" (UID: \"be9a9975-2bf4-47b3-918f-6ef9e4a88bca\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mdjkm" Nov 03 01:41:19.204798 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:41:19.204700 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b4a66c8-89c8-4894-9244-2a0c3587a06c-workload-socket\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xb7x2\" (UID: \"4b4a66c8-89c8-4894-9244-2a0c3587a06c\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xb7x2" Nov 03 01:41:19.204798 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:41:19.204726 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7srjg\" (UniqueName: \"kubernetes.io/projected/4b4a66c8-89c8-4894-9244-2a0c3587a06c-kube-api-access-7srjg\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xb7x2\" (UID: \"4b4a66c8-89c8-4894-9244-2a0c3587a06c\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xb7x2" Nov 03 01:41:19.204798 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:41:19.204762 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be9a9975-2bf4-47b3-918f-6ef9e4a88bca-workload-certs\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mdjkm\" (UID: \"be9a9975-2bf4-47b3-918f-6ef9e4a88bca\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mdjkm" Nov 03 01:41:19.204961 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:41:19.204893 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be9a9975-2bf4-47b3-918f-6ef9e4a88bca-workload-socket\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mdjkm\" (UID: \"be9a9975-2bf4-47b3-918f-6ef9e4a88bca\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mdjkm" Nov 03 01:41:19.204961 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:41:19.204944 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b4a66c8-89c8-4894-9244-2a0c3587a06c-credential-socket\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xb7x2\" (UID: \"4b4a66c8-89c8-4894-9244-2a0c3587a06c\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xb7x2" Nov 03 01:41:19.205043 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:41:19.204974 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be9a9975-2bf4-47b3-918f-6ef9e4a88bca-credential-socket\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mdjkm\" (UID: \"be9a9975-2bf4-47b3-918f-6ef9e4a88bca\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mdjkm" Nov 03 01:41:19.205043 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:41:19.204992 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b4a66c8-89c8-4894-9244-2a0c3587a06c-workload-certs\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xb7x2\" (UID: \"4b4a66c8-89c8-4894-9244-2a0c3587a06c\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xb7x2" Nov 03 01:41:19.205043 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:41:19.205026 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b4a66c8-89c8-4894-9244-2a0c3587a06c-istio-token\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xb7x2\" (UID: \"4b4a66c8-89c8-4894-9244-2a0c3587a06c\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xb7x2" Nov 03 01:41:19.205166 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:41:19.205047 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be9a9975-2bf4-47b3-918f-6ef9e4a88bca-workload-certs\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mdjkm\" (UID: \"be9a9975-2bf4-47b3-918f-6ef9e4a88bca\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mdjkm" Nov 03 01:41:19.205166 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:41:19.205059 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be9a9975-2bf4-47b3-918f-6ef9e4a88bca-istio-envoy\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mdjkm\" (UID: \"be9a9975-2bf4-47b3-918f-6ef9e4a88bca\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mdjkm" Nov 03 01:41:19.205166 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:41:19.205102 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be9a9975-2bf4-47b3-918f-6ef9e4a88bca-istio-data\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mdjkm\" (UID: \"be9a9975-2bf4-47b3-918f-6ef9e4a88bca\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mdjkm" Nov 03 01:41:19.205270 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:41:19.205172 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b4a66c8-89c8-4894-9244-2a0c3587a06c-istio-data\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xb7x2\" (UID: \"4b4a66c8-89c8-4894-9244-2a0c3587a06c\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xb7x2" Nov 03 01:41:19.205308 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:41:19.205277 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b4a66c8-89c8-4894-9244-2a0c3587a06c-istio-podinfo\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xb7x2\" (UID: \"4b4a66c8-89c8-4894-9244-2a0c3587a06c\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xb7x2" Nov 03 01:41:19.205355 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:41:19.205329 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be9a9975-2bf4-47b3-918f-6ef9e4a88bca-istio-token\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mdjkm\" (UID: \"be9a9975-2bf4-47b3-918f-6ef9e4a88bca\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mdjkm" Nov 03 01:41:19.205459 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:41:19.205397 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be9a9975-2bf4-47b3-918f-6ef9e4a88bca-istiod-ca-cert\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mdjkm\" (UID: \"be9a9975-2bf4-47b3-918f-6ef9e4a88bca\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mdjkm" Nov 03 01:41:19.205459 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:41:19.205423 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be9a9975-2bf4-47b3-918f-6ef9e4a88bca-istio-data\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mdjkm\" (UID: \"be9a9975-2bf4-47b3-918f-6ef9e4a88bca\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mdjkm" Nov 03 01:41:19.205459 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:41:19.205436 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b4a66c8-89c8-4894-9244-2a0c3587a06c-istiod-ca-cert\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xb7x2\" (UID: \"4b4a66c8-89c8-4894-9244-2a0c3587a06c\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xb7x2" Nov 03 01:41:19.205612 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:41:19.205490 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-464nr\" (UniqueName: \"kubernetes.io/projected/be9a9975-2bf4-47b3-918f-6ef9e4a88bca-kube-api-access-464nr\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mdjkm\" (UID: \"be9a9975-2bf4-47b3-918f-6ef9e4a88bca\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mdjkm" Nov 03 01:41:19.205950 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:41:19.205928 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be9a9975-2bf4-47b3-918f-6ef9e4a88bca-istiod-ca-cert\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mdjkm\" (UID: \"be9a9975-2bf4-47b3-918f-6ef9e4a88bca\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mdjkm" Nov 03 01:41:19.207288 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:41:19.207259 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be9a9975-2bf4-47b3-918f-6ef9e4a88bca-istio-envoy\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mdjkm\" (UID: \"be9a9975-2bf4-47b3-918f-6ef9e4a88bca\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mdjkm" Nov 03 01:41:19.207538 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:41:19.207519 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be9a9975-2bf4-47b3-918f-6ef9e4a88bca-istio-podinfo\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mdjkm\" (UID: \"be9a9975-2bf4-47b3-918f-6ef9e4a88bca\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mdjkm" Nov 03 01:41:19.269577 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:41:19.269539 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be9a9975-2bf4-47b3-918f-6ef9e4a88bca-istio-token\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mdjkm\" (UID: \"be9a9975-2bf4-47b3-918f-6ef9e4a88bca\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mdjkm" Nov 03 01:41:19.269769 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:41:19.269651 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-464nr\" (UniqueName: \"kubernetes.io/projected/be9a9975-2bf4-47b3-918f-6ef9e4a88bca-kube-api-access-464nr\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mdjkm\" (UID: \"be9a9975-2bf4-47b3-918f-6ef9e4a88bca\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mdjkm" Nov 03 01:41:19.306956 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:41:19.306913 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b4a66c8-89c8-4894-9244-2a0c3587a06c-istio-data\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xb7x2\" (UID: \"4b4a66c8-89c8-4894-9244-2a0c3587a06c\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xb7x2" Nov 03 01:41:19.307145 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:41:19.306977 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b4a66c8-89c8-4894-9244-2a0c3587a06c-istio-podinfo\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xb7x2\" (UID: \"4b4a66c8-89c8-4894-9244-2a0c3587a06c\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xb7x2" Nov 03 01:41:19.307145 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:41:19.307009 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b4a66c8-89c8-4894-9244-2a0c3587a06c-istiod-ca-cert\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xb7x2\" (UID: \"4b4a66c8-89c8-4894-9244-2a0c3587a06c\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xb7x2" Nov 03 01:41:19.307145 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:41:19.307097 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b4a66c8-89c8-4894-9244-2a0c3587a06c-istio-envoy\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xb7x2\" (UID: \"4b4a66c8-89c8-4894-9244-2a0c3587a06c\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xb7x2" Nov 03 01:41:19.307276 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:41:19.307186 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b4a66c8-89c8-4894-9244-2a0c3587a06c-workload-socket\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xb7x2\" (UID: \"4b4a66c8-89c8-4894-9244-2a0c3587a06c\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xb7x2" Nov 03 01:41:19.307276 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:41:19.307203 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7srjg\" (UniqueName: \"kubernetes.io/projected/4b4a66c8-89c8-4894-9244-2a0c3587a06c-kube-api-access-7srjg\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xb7x2\" (UID: \"4b4a66c8-89c8-4894-9244-2a0c3587a06c\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xb7x2" Nov 03 01:41:19.307276 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:41:19.307251 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b4a66c8-89c8-4894-9244-2a0c3587a06c-credential-socket\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xb7x2\" (UID: \"4b4a66c8-89c8-4894-9244-2a0c3587a06c\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xb7x2" Nov 03 01:41:19.307276 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:41:19.307270 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b4a66c8-89c8-4894-9244-2a0c3587a06c-workload-certs\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xb7x2\" (UID: \"4b4a66c8-89c8-4894-9244-2a0c3587a06c\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xb7x2" Nov 03 01:41:19.307523 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:41:19.307290 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b4a66c8-89c8-4894-9244-2a0c3587a06c-istio-token\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xb7x2\" (UID: \"4b4a66c8-89c8-4894-9244-2a0c3587a06c\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xb7x2" Nov 03 01:41:19.307685 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:41:19.307662 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b4a66c8-89c8-4894-9244-2a0c3587a06c-workload-socket\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xb7x2\" (UID: \"4b4a66c8-89c8-4894-9244-2a0c3587a06c\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xb7x2" Nov 03 01:41:19.307767 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:41:19.307697 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b4a66c8-89c8-4894-9244-2a0c3587a06c-credential-socket\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xb7x2\" (UID: \"4b4a66c8-89c8-4894-9244-2a0c3587a06c\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xb7x2" Nov 03 01:41:19.307917 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:41:19.307894 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b4a66c8-89c8-4894-9244-2a0c3587a06c-workload-certs\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xb7x2\" (UID: \"4b4a66c8-89c8-4894-9244-2a0c3587a06c\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xb7x2" Nov 03 01:41:19.307992 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:41:19.307978 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b4a66c8-89c8-4894-9244-2a0c3587a06c-istio-data\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xb7x2\" (UID: \"4b4a66c8-89c8-4894-9244-2a0c3587a06c\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xb7x2" Nov 03 01:41:19.308043 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:41:19.307980 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b4a66c8-89c8-4894-9244-2a0c3587a06c-istiod-ca-cert\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xb7x2\" (UID: \"4b4a66c8-89c8-4894-9244-2a0c3587a06c\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xb7x2" Nov 03 01:41:19.309710 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:41:19.309685 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b4a66c8-89c8-4894-9244-2a0c3587a06c-istio-podinfo\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xb7x2\" (UID: \"4b4a66c8-89c8-4894-9244-2a0c3587a06c\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xb7x2" Nov 03 01:41:19.309795 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:41:19.309716 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b4a66c8-89c8-4894-9244-2a0c3587a06c-istio-envoy\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xb7x2\" (UID: \"4b4a66c8-89c8-4894-9244-2a0c3587a06c\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xb7x2" Nov 03 01:41:19.313981 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:41:19.313948 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mdjkm" Nov 03 01:41:19.371068 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:41:19.371026 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b4a66c8-89c8-4894-9244-2a0c3587a06c-istio-token\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xb7x2\" (UID: \"4b4a66c8-89c8-4894-9244-2a0c3587a06c\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xb7x2" Nov 03 01:41:19.371608 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:41:19.371571 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7srjg\" (UniqueName: \"kubernetes.io/projected/4b4a66c8-89c8-4894-9244-2a0c3587a06c-kube-api-access-7srjg\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xb7x2\" (UID: \"4b4a66c8-89c8-4894-9244-2a0c3587a06c\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xb7x2" Nov 03 01:41:19.660533 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:41:19.660485 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xb7x2" Nov 03 01:41:19.752659 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:41:19.752619 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mdjkm"] Nov 03 01:41:19.754715 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:41:19.754671 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe9a9975_2bf4_47b3_918f_6ef9e4a88bca.slice/crio-e2a5ced8093ae060ec1f953bd67da6cf0ce74bc93dd3f0b48cf3b76bb6395855 WatchSource:0}: Error finding container e2a5ced8093ae060ec1f953bd67da6cf0ce74bc93dd3f0b48cf3b76bb6395855: Status 404 returned error can't find the container with id e2a5ced8093ae060ec1f953bd67da6cf0ce74bc93dd3f0b48cf3b76bb6395855 Nov 03 01:41:19.763307 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:41:19.763259 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 01:41:19.763527 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:41:19.763354 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 01:41:19.763527 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:41:19.763448 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 01:41:20.040590 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:41:20.040550 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b4a66c8_89c8_4894_9244_2a0c3587a06c.slice/crio-ab06b1f56962851c9432108a755020631ba9611a982c469c52f22c1e5013b888 WatchSource:0}: Error finding container ab06b1f56962851c9432108a755020631ba9611a982c469c52f22c1e5013b888: Status 404 returned error can't find the container with id ab06b1f56962851c9432108a755020631ba9611a982c469c52f22c1e5013b888 Nov 03 01:41:20.043077 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:41:20.043051 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xb7x2"] Nov 03 01:41:20.043688 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:41:20.043659 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 01:41:20.043737 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:41:20.043723 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 01:41:20.043782 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:41:20.043752 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 01:41:20.781480 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:41:20.779207 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xb7x2" event={"ID":"4b4a66c8-89c8-4894-9244-2a0c3587a06c","Type":"ContainerStarted","Data":"7135cfd55c1c212acf25acdbe635f8de4242b4df092daf702f82eb40fe4ac803"} Nov 03 01:41:20.781480 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:41:20.779276 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xb7x2" event={"ID":"4b4a66c8-89c8-4894-9244-2a0c3587a06c","Type":"ContainerStarted","Data":"ab06b1f56962851c9432108a755020631ba9611a982c469c52f22c1e5013b888"} Nov 03 01:41:20.782602 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:41:20.782561 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mdjkm" event={"ID":"be9a9975-2bf4-47b3-918f-6ef9e4a88bca","Type":"ContainerStarted","Data":"0e156e7450453f8c7398bec9f3b615389046481461ea0840f94f806e34ff864d"} Nov 03 01:41:20.782752 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:41:20.782632 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mdjkm" event={"ID":"be9a9975-2bf4-47b3-918f-6ef9e4a88bca","Type":"ContainerStarted","Data":"e2a5ced8093ae060ec1f953bd67da6cf0ce74bc93dd3f0b48cf3b76bb6395855"} Nov 03 01:41:20.908594 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:41:20.908532 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xb7x2" podStartSLOduration=2.908514972 podStartE2EDuration="2.908514972s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:41:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:41:20.906964796 +0000 UTC m=+1362.936447088" watchObservedRunningTime="2026-11-03 01:41:20.908514972 +0000 UTC m=+1362.937997329" Nov 03 01:41:21.055462 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:41:21.041070 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mdjkm" podStartSLOduration=3.041050012 podStartE2EDuration="3.041050012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:41:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:41:21.04031451 +0000 UTC m=+1363.069796816" watchObservedRunningTime="2026-11-03 01:41:21.041050012 +0000 UTC m=+1363.070532309" Nov 03 01:41:21.314605 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:41:21.314510 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mdjkm" Nov 03 01:41:21.316302 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:41:21.316270 5437 patch_prober.go:29] interesting pod/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mdjkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 03 01:41:21.316515 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:41:21.316331 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mdjkm" podUID="be9a9975-2bf4-47b3-918f-6ef9e4a88bca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 03 01:41:21.663422 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:41:21.663273 5437 patch_prober.go:29] interesting pod/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xb7x2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Nov 03 01:41:21.663707 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:41:21.663673 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xb7x2" podUID="4b4a66c8-89c8-4894-9244-2a0c3587a06c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Nov 03 01:41:21.664164 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:41:21.664136 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xb7x2" Nov 03 01:41:22.314569 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:41:22.314528 5437 patch_prober.go:29] interesting pod/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mdjkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 03 01:41:22.314997 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:41:22.314592 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mdjkm" podUID="be9a9975-2bf4-47b3-918f-6ef9e4a88bca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 03 01:41:22.664456 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:41:22.663849 5437 patch_prober.go:29] interesting pod/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xb7x2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Nov 03 01:41:22.664456 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:41:22.663924 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xb7x2" podUID="4b4a66c8-89c8-4894-9244-2a0c3587a06c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Nov 03 01:41:23.314732 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:41:23.314692 5437 patch_prober.go:29] interesting pod/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mdjkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 03 01:41:23.315164 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:41:23.314751 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mdjkm" podUID="be9a9975-2bf4-47b3-918f-6ef9e4a88bca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 03 01:41:23.662458 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:41:23.661809 5437 patch_prober.go:29] interesting pod/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xb7x2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Nov 03 01:41:23.662458 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:41:23.661883 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xb7x2" podUID="4b4a66c8-89c8-4894-9244-2a0c3587a06c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Nov 03 01:41:23.932593 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:41:23.932491 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:24.314633 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:41:24.314596 5437 patch_prober.go:29] interesting pod/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mdjkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 03 01:41:24.314884 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:41:24.314860 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mdjkm" podUID="be9a9975-2bf4-47b3-918f-6ef9e4a88bca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 03 01:41:24.668207 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:41:24.667384 5437 patch_prober.go:29] interesting pod/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xb7x2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Nov 03 01:41:24.668207 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:41:24.667458 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xb7x2" podUID="4b4a66c8-89c8-4894-9244-2a0c3587a06c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Nov 03 01:41:25.314849 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:41:25.314813 5437 patch_prober.go:29] interesting pod/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mdjkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 03 01:41:25.315119 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:41:25.314874 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mdjkm" podUID="be9a9975-2bf4-47b3-918f-6ef9e4a88bca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 03 01:41:25.665927 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:41:25.665822 5437 patch_prober.go:29] interesting pod/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xb7x2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Nov 03 01:41:25.666233 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:41:25.666195 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xb7x2" podUID="4b4a66c8-89c8-4894-9244-2a0c3587a06c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Nov 03 01:41:26.315005 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:41:26.314967 5437 patch_prober.go:29] interesting pod/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mdjkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 03 01:41:26.315191 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:41:26.315060 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mdjkm" podUID="be9a9975-2bf4-47b3-918f-6ef9e4a88bca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 03 01:41:26.660929 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:41:26.660839 5437 patch_prober.go:29] interesting pod/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xb7x2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Nov 03 01:41:26.660929 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:41:26.660913 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xb7x2" podUID="4b4a66c8-89c8-4894-9244-2a0c3587a06c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Nov 03 01:41:27.372541 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:41:27.372505 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mdjkm" Nov 03 01:41:27.373489 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:41:27.373467 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mdjkm" Nov 03 01:41:27.374192 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:41:27.374171 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mdjkm" Nov 03 01:41:27.699807 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:41:27.698430 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xb7x2" Nov 03 01:41:27.700740 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:41:27.700277 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xb7x2" Nov 03 01:41:27.701140 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:41:27.701115 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xb7x2" Nov 03 01:41:28.801071 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:41:28.801034 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:29.157291 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:41:29.157034 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:29.978326 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:41:29.978281 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:30.842111 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:41:30.842070 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:31.081436 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:41:31.081345 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:33.068513 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:41:33.068461 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:33.892538 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:41:33.892492 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:35.566775 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:41:35.566732 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:35.771782 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:41:35.771740 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:38.714271 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:41:38.714235 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:38.963091 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:41:38.963054 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:39.197391 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:41:39.194533 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:40.208296 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:41:40.208252 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:40.413786 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:41:40.413746 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:43.456265 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:41:43.456229 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:44.035297 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:41:44.035259 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 01:41:44.069336 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:41:44.069292 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:44.272343 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:41:44.272307 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:48.811540 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:41:48.811475 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:53.573109 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:41:53.573069 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:55.371516 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:41:55.371480 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:55.627667 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:41:55.627565 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:58.318930 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:41:58.318878 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:03.087344 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:42:03.087301 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:07.845213 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:42:07.845170 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:08.055061 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:42:08.055014 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:08.290050 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:42:08.289955 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:10.082999 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:42:10.082963 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:13.172583 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:42:13.172543 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:14.037020 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:42:14.036983 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 01:42:15.970651 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:42:15.970605 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:17.671289 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:42:17.671250 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:22.484162 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:42:22.484117 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:27.603989 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:42:27.603948 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:27.810505 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:42:27.810465 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:28.183754 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:42:28.183712 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:28.560789 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:42:28.560755 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:32.522560 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:42:32.522518 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:32.846008 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:42:32.845941 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:33.176120 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:42:33.176022 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:33.395295 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:42:33.395256 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:37.691325 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:42:37.691268 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:40.681419 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:42:40.681382 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:40.932995 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:42:40.932903 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:42.498002 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:42:42.497960 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:43.796151 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:42:43.796115 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:44.054553 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:42:44.054514 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 01:42:47.236765 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:42:47.236682 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:51.934492 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:42:51.934448 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:52.155447 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:42:52.155403 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:52.560265 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:42:52.559947 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:52.782933 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:42:52.782892 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:56.365564 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:42:56.365524 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:56.621422 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:42:56.621296 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:57.177565 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:42:57.177527 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:57.388085 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:42:57.388047 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:57.909434 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:42:57.909399 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:58.498662 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:42:58.498625 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:58.840781 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:42:58.840742 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:59.499551 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:42:59.499507 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:59.706005 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:42:59.705958 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:43:02.647675 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:43:02.647635 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:43:03.232459 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:43:03.232416 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:43:03.486669 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:43:03.486571 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:43:06.401218 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:43:06.401179 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:43:06.654881 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:43:06.654774 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:43:08.032104 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:43:08.032051 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:43:13.007198 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:43:13.007157 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-948470-78b87884df-7hp5g"] Nov 03 01:43:13.011576 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:43:13.011547 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-948470-78b87884df-7hp5g" Nov 03 01:43:13.109236 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:43:13.109189 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3361c0e5-85a7-466e-8c60-024f958e7227-openshift-service-ca\") pod \"maas-api-e2e-api-a-948470-78b87884df-7hp5g\" (UID: \"3361c0e5-85a7-466e-8c60-024f958e7227\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-948470-78b87884df-7hp5g" Nov 03 01:43:13.109457 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:43:13.109252 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3361c0e5-85a7-466e-8c60-024f958e7227-maas-api-tls\") pod \"maas-api-e2e-api-a-948470-78b87884df-7hp5g\" (UID: \"3361c0e5-85a7-466e-8c60-024f958e7227\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-948470-78b87884df-7hp5g" Nov 03 01:43:13.109457 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:43:13.109348 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8966\" (UniqueName: \"kubernetes.io/projected/3361c0e5-85a7-466e-8c60-024f958e7227-kube-api-access-z8966\") pod \"maas-api-e2e-api-a-948470-78b87884df-7hp5g\" (UID: \"3361c0e5-85a7-466e-8c60-024f958e7227\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-948470-78b87884df-7hp5g" Nov 03 01:43:13.109582 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:43:13.109554 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3361c0e5-85a7-466e-8c60-024f958e7227-metrics-certs\") pod \"maas-api-e2e-api-a-948470-78b87884df-7hp5g\" (UID: \"3361c0e5-85a7-466e-8c60-024f958e7227\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-948470-78b87884df-7hp5g" Nov 03 01:43:13.119632 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:43:13.119599 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-948470-serving-cert\"" Nov 03 01:43:13.122765 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:43:13.122732 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-948470-78b87884df-7hp5g"] Nov 03 01:43:13.210509 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:43:13.210468 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3361c0e5-85a7-466e-8c60-024f958e7227-openshift-service-ca\") pod \"maas-api-e2e-api-a-948470-78b87884df-7hp5g\" (UID: \"3361c0e5-85a7-466e-8c60-024f958e7227\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-948470-78b87884df-7hp5g" Nov 03 01:43:13.210705 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:43:13.210536 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3361c0e5-85a7-466e-8c60-024f958e7227-maas-api-tls\") pod \"maas-api-e2e-api-a-948470-78b87884df-7hp5g\" (UID: \"3361c0e5-85a7-466e-8c60-024f958e7227\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-948470-78b87884df-7hp5g" Nov 03 01:43:13.210705 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:43:13.210564 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8966\" (UniqueName: \"kubernetes.io/projected/3361c0e5-85a7-466e-8c60-024f958e7227-kube-api-access-z8966\") pod \"maas-api-e2e-api-a-948470-78b87884df-7hp5g\" (UID: \"3361c0e5-85a7-466e-8c60-024f958e7227\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-948470-78b87884df-7hp5g" Nov 03 01:43:13.210705 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:43:13.210648 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3361c0e5-85a7-466e-8c60-024f958e7227-metrics-certs\") pod \"maas-api-e2e-api-a-948470-78b87884df-7hp5g\" (UID: \"3361c0e5-85a7-466e-8c60-024f958e7227\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-948470-78b87884df-7hp5g" Nov 03 01:43:13.210705 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:43:13.210698 5437 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-948470-serving-cert: secret "maas-api-e2e-api-a-948470-serving-cert" not found Nov 03 01:43:13.211042 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:43:13.210783 5437 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3361c0e5-85a7-466e-8c60-024f958e7227-maas-api-tls podName:3361c0e5-85a7-466e-8c60-024f958e7227 nodeName:}" failed. No retries permitted until 2026-11-03 01:43:13.710761692 +0000 UTC m=+1475.740243980 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3361c0e5-85a7-466e-8c60-024f958e7227-maas-api-tls") pod "maas-api-e2e-api-a-948470-78b87884df-7hp5g" (UID: "3361c0e5-85a7-466e-8c60-024f958e7227") : secret "maas-api-e2e-api-a-948470-serving-cert" not found Nov 03 01:43:13.211437 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:43:13.211417 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3361c0e5-85a7-466e-8c60-024f958e7227-openshift-service-ca\") pod \"maas-api-e2e-api-a-948470-78b87884df-7hp5g\" (UID: \"3361c0e5-85a7-466e-8c60-024f958e7227\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-948470-78b87884df-7hp5g" Nov 03 01:43:13.213230 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:43:13.213208 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3361c0e5-85a7-466e-8c60-024f958e7227-metrics-certs\") pod \"maas-api-e2e-api-a-948470-78b87884df-7hp5g\" (UID: \"3361c0e5-85a7-466e-8c60-024f958e7227\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-948470-78b87884df-7hp5g" Nov 03 01:43:13.274457 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:43:13.274359 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8966\" (UniqueName: \"kubernetes.io/projected/3361c0e5-85a7-466e-8c60-024f958e7227-kube-api-access-z8966\") pod \"maas-api-e2e-api-a-948470-78b87884df-7hp5g\" (UID: \"3361c0e5-85a7-466e-8c60-024f958e7227\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-948470-78b87884df-7hp5g" Nov 03 01:43:13.295727 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:43:13.295691 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:43:13.503327 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:43:13.503286 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:43:13.707043 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:43:13.707008 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:43:13.718867 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:43:13.718831 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3361c0e5-85a7-466e-8c60-024f958e7227-maas-api-tls\") pod \"maas-api-e2e-api-a-948470-78b87884df-7hp5g\" (UID: \"3361c0e5-85a7-466e-8c60-024f958e7227\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-948470-78b87884df-7hp5g" Nov 03 01:43:13.721430 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:43:13.721356 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3361c0e5-85a7-466e-8c60-024f958e7227-maas-api-tls\") pod \"maas-api-e2e-api-a-948470-78b87884df-7hp5g\" (UID: \"3361c0e5-85a7-466e-8c60-024f958e7227\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-948470-78b87884df-7hp5g" Nov 03 01:43:13.927718 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:43:13.927679 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-948470-78b87884df-7hp5g" Nov 03 01:43:14.058478 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:43:14.058437 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 01:43:14.319216 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:43:14.319166 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3361c0e5_85a7_466e_8c60_024f958e7227.slice/crio-7fb28eda35b6df34f3a5d3d3b8a43d5a85e4a6a9dddbbb1454d75b8133c4b2a2 WatchSource:0}: Error finding container 7fb28eda35b6df34f3a5d3d3b8a43d5a85e4a6a9dddbbb1454d75b8133c4b2a2: Status 404 returned error can't find the container with id 7fb28eda35b6df34f3a5d3d3b8a43d5a85e4a6a9dddbbb1454d75b8133c4b2a2 Nov 03 01:43:14.321113 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:43:14.321089 5437 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 01:43:14.322083 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:43:14.322059 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-948470-78b87884df-7hp5g"] Nov 03 01:43:14.597061 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:43:14.596965 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-948470-78b87884df-7hp5g" event={"ID":"3361c0e5-85a7-466e-8c60-024f958e7227","Type":"ContainerStarted","Data":"7fb28eda35b6df34f3a5d3d3b8a43d5a85e4a6a9dddbbb1454d75b8133c4b2a2"} Nov 03 01:43:15.605650 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:43:15.605609 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-948470-78b87884df-7hp5g" event={"ID":"3361c0e5-85a7-466e-8c60-024f958e7227","Type":"ContainerStarted","Data":"5b6ec2f29033672dbd525b113db8e28fda3caac8b82e296d087199d828c666b7"} Nov 03 01:43:15.606078 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:43:15.605819 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-948470-78b87884df-7hp5g" Nov 03 01:43:15.732818 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:43:15.732754 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-948470-78b87884df-7hp5g" podStartSLOduration=2.777647321 podStartE2EDuration="3.732736107s" totalImagesPullingTime="955.088786ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:43:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:43:15.729728379 +0000 UTC m=+1477.759210679" watchObservedRunningTime="2026-11-03 01:43:15.732736107 +0000 UTC m=+1477.762218401" Nov 03 01:43:18.547257 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:43:18.545901 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:43:21.618440 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:43:21.618407 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-948470-78b87884df-7hp5g" Nov 03 01:43:23.337662 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:43:23.337623 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:43:28.088168 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:43:28.088133 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:43:32.858539 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:43:32.858495 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:43:37.613486 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:43:37.613378 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:43:40.912187 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:43:40.912148 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:43:41.560063 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:43:41.560005 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:43:41.762751 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:43:41.762715 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:43:42.066208 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:43:42.066171 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5e048b-764bb84787-mbchq"] Nov 03 01:43:42.070763 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:43:42.070740 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5e048b-764bb84787-mbchq" Nov 03 01:43:42.126289 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:43:42.126249 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5e048b-764bb84787-mbchq"] Nov 03 01:43:42.173317 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:43:42.173278 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-5e048b-serving-cert\"" Nov 03 01:43:42.220094 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:43:42.220054 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8ce6d38-37d9-45ef-a6c0-4f7ad8733998-maas-api-tls\") pod \"maas-api-e2e-api-a-5e048b-764bb84787-mbchq\" (UID: \"b8ce6d38-37d9-45ef-a6c0-4f7ad8733998\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5e048b-764bb84787-mbchq" Nov 03 01:43:42.220304 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:43:42.220232 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzzxq\" (UniqueName: \"kubernetes.io/projected/b8ce6d38-37d9-45ef-a6c0-4f7ad8733998-kube-api-access-wzzxq\") pod \"maas-api-e2e-api-a-5e048b-764bb84787-mbchq\" (UID: \"b8ce6d38-37d9-45ef-a6c0-4f7ad8733998\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5e048b-764bb84787-mbchq" Nov 03 01:43:42.220510 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:43:42.220482 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8ce6d38-37d9-45ef-a6c0-4f7ad8733998-metrics-certs\") pod \"maas-api-e2e-api-a-5e048b-764bb84787-mbchq\" (UID: \"b8ce6d38-37d9-45ef-a6c0-4f7ad8733998\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5e048b-764bb84787-mbchq" Nov 03 01:43:42.220588 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:43:42.220543 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8ce6d38-37d9-45ef-a6c0-4f7ad8733998-openshift-service-ca\") pod \"maas-api-e2e-api-a-5e048b-764bb84787-mbchq\" (UID: \"b8ce6d38-37d9-45ef-a6c0-4f7ad8733998\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5e048b-764bb84787-mbchq" Nov 03 01:43:42.323066 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:43:42.322970 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:43:42.323357 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:43:42.323321 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzzxq\" (UniqueName: \"kubernetes.io/projected/b8ce6d38-37d9-45ef-a6c0-4f7ad8733998-kube-api-access-wzzxq\") pod \"maas-api-e2e-api-a-5e048b-764bb84787-mbchq\" (UID: \"b8ce6d38-37d9-45ef-a6c0-4f7ad8733998\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5e048b-764bb84787-mbchq" Nov 03 01:43:42.323647 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:43:42.323625 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8ce6d38-37d9-45ef-a6c0-4f7ad8733998-metrics-certs\") pod \"maas-api-e2e-api-a-5e048b-764bb84787-mbchq\" (UID: \"b8ce6d38-37d9-45ef-a6c0-4f7ad8733998\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5e048b-764bb84787-mbchq" Nov 03 01:43:42.323779 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:43:42.323680 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8ce6d38-37d9-45ef-a6c0-4f7ad8733998-openshift-service-ca\") pod \"maas-api-e2e-api-a-5e048b-764bb84787-mbchq\" (UID: \"b8ce6d38-37d9-45ef-a6c0-4f7ad8733998\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5e048b-764bb84787-mbchq" Nov 03 01:43:42.323779 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:43:42.323731 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8ce6d38-37d9-45ef-a6c0-4f7ad8733998-maas-api-tls\") pod \"maas-api-e2e-api-a-5e048b-764bb84787-mbchq\" (UID: \"b8ce6d38-37d9-45ef-a6c0-4f7ad8733998\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5e048b-764bb84787-mbchq" Nov 03 01:43:42.324468 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:43:42.323928 5437 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-5e048b-serving-cert: secret "maas-api-e2e-api-a-5e048b-serving-cert" not found Nov 03 01:43:42.324468 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:43:42.324010 5437 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8ce6d38-37d9-45ef-a6c0-4f7ad8733998-maas-api-tls podName:b8ce6d38-37d9-45ef-a6c0-4f7ad8733998 nodeName:}" failed. No retries permitted until 2026-11-03 01:43:42.823987361 +0000 UTC m=+1504.853469641 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b8ce6d38-37d9-45ef-a6c0-4f7ad8733998-maas-api-tls") pod "maas-api-e2e-api-a-5e048b-764bb84787-mbchq" (UID: "b8ce6d38-37d9-45ef-a6c0-4f7ad8733998") : secret "maas-api-e2e-api-a-5e048b-serving-cert" not found Nov 03 01:43:42.325102 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:43:42.325065 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8ce6d38-37d9-45ef-a6c0-4f7ad8733998-openshift-service-ca\") pod \"maas-api-e2e-api-a-5e048b-764bb84787-mbchq\" (UID: \"b8ce6d38-37d9-45ef-a6c0-4f7ad8733998\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5e048b-764bb84787-mbchq" Nov 03 01:43:42.329205 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:43:42.329178 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8ce6d38-37d9-45ef-a6c0-4f7ad8733998-metrics-certs\") pod \"maas-api-e2e-api-a-5e048b-764bb84787-mbchq\" (UID: \"b8ce6d38-37d9-45ef-a6c0-4f7ad8733998\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5e048b-764bb84787-mbchq" Nov 03 01:43:42.387569 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:43:42.387530 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzzxq\" (UniqueName: \"kubernetes.io/projected/b8ce6d38-37d9-45ef-a6c0-4f7ad8733998-kube-api-access-wzzxq\") pod \"maas-api-e2e-api-a-5e048b-764bb84787-mbchq\" (UID: \"b8ce6d38-37d9-45ef-a6c0-4f7ad8733998\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5e048b-764bb84787-mbchq" Nov 03 01:43:42.529400 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:43:42.529336 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:43:42.734821 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:43:42.734736 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:43:42.831056 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:43:42.831017 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8ce6d38-37d9-45ef-a6c0-4f7ad8733998-maas-api-tls\") pod \"maas-api-e2e-api-a-5e048b-764bb84787-mbchq\" (UID: \"b8ce6d38-37d9-45ef-a6c0-4f7ad8733998\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5e048b-764bb84787-mbchq" Nov 03 01:43:42.833683 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:43:42.833648 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8ce6d38-37d9-45ef-a6c0-4f7ad8733998-maas-api-tls\") pod \"maas-api-e2e-api-a-5e048b-764bb84787-mbchq\" (UID: \"b8ce6d38-37d9-45ef-a6c0-4f7ad8733998\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5e048b-764bb84787-mbchq" Nov 03 01:43:42.985734 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:43:42.985640 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5e048b-764bb84787-mbchq" Nov 03 01:43:43.363699 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:43:43.363659 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8ce6d38_37d9_45ef_a6c0_4f7ad8733998.slice/crio-745071cfa17021c505a83ba3449479dd6f1eb3e4e4e9528bf5284e1a5458a1f6 WatchSource:0}: Error finding container 745071cfa17021c505a83ba3449479dd6f1eb3e4e4e9528bf5284e1a5458a1f6: Status 404 returned error can't find the container with id 745071cfa17021c505a83ba3449479dd6f1eb3e4e4e9528bf5284e1a5458a1f6 Nov 03 01:43:43.367469 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:43:43.367439 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5e048b-764bb84787-mbchq"] Nov 03 01:43:43.817203 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:43:43.817164 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5e048b-764bb84787-mbchq" event={"ID":"b8ce6d38-37d9-45ef-a6c0-4f7ad8733998","Type":"ContainerStarted","Data":"745071cfa17021c505a83ba3449479dd6f1eb3e4e4e9528bf5284e1a5458a1f6"} Nov 03 01:43:44.036820 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:43:44.036781 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 01:43:44.124579 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:43:44.124474 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:43:44.825127 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:43:44.825085 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5e048b-764bb84787-mbchq" event={"ID":"b8ce6d38-37d9-45ef-a6c0-4f7ad8733998","Type":"ContainerStarted","Data":"4d71551d6bbb8b51a1ee0b568e7ab2d7e7efb8fd93963b2d61fae7d703852812"} Nov 03 01:43:44.825632 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:43:44.825161 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5e048b-764bb84787-mbchq" Nov 03 01:43:44.948563 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:43:44.948510 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5e048b-764bb84787-mbchq" podStartSLOduration=2.104747291 podStartE2EDuration="2.948494887s" totalImagesPullingTime="843.747596ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:43:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:43:44.945634928 +0000 UTC m=+1506.975117218" watchObservedRunningTime="2026-11-03 01:43:44.948494887 +0000 UTC m=+1506.977977181" Nov 03 01:43:45.869056 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:43:45.869005 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:43:46.095485 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:43:46.095445 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:43:47.565576 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:43:47.561110 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:43:50.414310 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:43:50.414275 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:43:50.615560 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:43:50.615463 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:43:50.839458 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:43:50.839415 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5e048b-764bb84787-mbchq" Nov 03 01:43:52.521172 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:43:52.521135 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:43:52.725123 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:43:52.725083 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:43:52.929829 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:43:52.929794 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:43:57.919257 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:43:57.919203 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:44:03.069151 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:44:03.069088 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:44:08.289227 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:44:08.289172 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:44:09.650935 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:44:09.650888 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:44:09.849807 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:44:09.849768 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:44:10.165223 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:44:10.165173 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2b9f22c1-7cd5c774dc-xbtwr"] Nov 03 01:44:10.479729 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:44:10.479611 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:44:10.678276 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:44:10.678230 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:44:13.194255 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:44:13.194213 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:44:14.040826 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:44:14.040783 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 01:44:15.165789 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:44:15.165723 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-2b9f22c1-7cd5c774dc-xbtwr" podUID="18277241-9019-4b65-86b7-7084af0b742d" containerName="payload-pre-processing" containerID="cri-o://ea4323a5a1d4f1e24a8d1e202b285fb2f3b47673dbb910e48df50b1e9ec430cd" gracePeriod=25 Nov 03 01:44:15.250667 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:44:15.250623 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:44:16.014968 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:44:16.014924 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:44:16.638305 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:44:16.638265 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:44:16.966569 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:44:16.966429 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:44:18.573429 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:44:18.573355 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:44:21.810940 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:44:21.810896 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:44:21.952822 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:44:21.952780 5437 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-2b9f22c1-7cd5c774dc-xbtwr" podUID="18277241-9019-4b65-86b7-7084af0b742d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:44:22.059935 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:44:22.059896 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:44:23.749157 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:44:23.749117 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:44:24.975191 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:44:24.975152 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:44:25.231120 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:44:25.231019 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:44:25.487251 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:44:25.487150 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4952b1-ccfb5b59f-4r2rj"] Nov 03 01:44:25.492748 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:44:25.492718 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-4952b1-ccfb5b59f-4r2rj" Nov 03 01:44:25.503516 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:44:25.494324 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6faa241e-9510-45af-9e59-e3aa35524217-plugins-config-volume\") pod \"payload-processing-e2e-api-b-4952b1-ccfb5b59f-4r2rj\" (UID: \"6faa241e-9510-45af-9e59-e3aa35524217\") " pod="openshift-ingress/payload-processing-e2e-api-b-4952b1-ccfb5b59f-4r2rj" Nov 03 01:44:25.503720 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:44:25.503634 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bg8mx\" (UniqueName: \"kubernetes.io/projected/6faa241e-9510-45af-9e59-e3aa35524217-kube-api-access-bg8mx\") pod \"payload-processing-e2e-api-b-4952b1-ccfb5b59f-4r2rj\" (UID: \"6faa241e-9510-45af-9e59-e3aa35524217\") " pod="openshift-ingress/payload-processing-e2e-api-b-4952b1-ccfb5b59f-4r2rj" Nov 03 01:44:25.547719 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:44:25.547679 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4952b1-ccfb5b59f-4r2rj"] Nov 03 01:44:25.597791 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:44:25.597750 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-4952b1\"" Nov 03 01:44:25.598476 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:44:25.598447 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-4952b1-dockercfg-w4r69\"" Nov 03 01:44:25.605421 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:44:25.605390 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bg8mx\" (UniqueName: \"kubernetes.io/projected/6faa241e-9510-45af-9e59-e3aa35524217-kube-api-access-bg8mx\") pod \"payload-processing-e2e-api-b-4952b1-ccfb5b59f-4r2rj\" (UID: \"6faa241e-9510-45af-9e59-e3aa35524217\") " pod="openshift-ingress/payload-processing-e2e-api-b-4952b1-ccfb5b59f-4r2rj" Nov 03 01:44:25.605654 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:44:25.605629 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6faa241e-9510-45af-9e59-e3aa35524217-plugins-config-volume\") pod \"payload-processing-e2e-api-b-4952b1-ccfb5b59f-4r2rj\" (UID: \"6faa241e-9510-45af-9e59-e3aa35524217\") " pod="openshift-ingress/payload-processing-e2e-api-b-4952b1-ccfb5b59f-4r2rj" Nov 03 01:44:25.606333 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:44:25.606316 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6faa241e-9510-45af-9e59-e3aa35524217-plugins-config-volume\") pod \"payload-processing-e2e-api-b-4952b1-ccfb5b59f-4r2rj\" (UID: \"6faa241e-9510-45af-9e59-e3aa35524217\") " pod="openshift-ingress/payload-processing-e2e-api-b-4952b1-ccfb5b59f-4r2rj" Nov 03 01:44:25.669183 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:44:25.669150 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bg8mx\" (UniqueName: \"kubernetes.io/projected/6faa241e-9510-45af-9e59-e3aa35524217-kube-api-access-bg8mx\") pod \"payload-processing-e2e-api-b-4952b1-ccfb5b59f-4r2rj\" (UID: \"6faa241e-9510-45af-9e59-e3aa35524217\") " pod="openshift-ingress/payload-processing-e2e-api-b-4952b1-ccfb5b59f-4r2rj" Nov 03 01:44:25.830618 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:44:25.830576 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-4952b1-ccfb5b59f-4r2rj" Nov 03 01:44:26.215249 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:44:26.215210 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6faa241e_9510_45af_9e59_e3aa35524217.slice/crio-782b07eaa4d43eda95fe4dda1c206b299adc6cf4e803fb90fc2f11f611ae4dea WatchSource:0}: Error finding container 782b07eaa4d43eda95fe4dda1c206b299adc6cf4e803fb90fc2f11f611ae4dea: Status 404 returned error can't find the container with id 782b07eaa4d43eda95fe4dda1c206b299adc6cf4e803fb90fc2f11f611ae4dea Nov 03 01:44:26.218842 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:44:26.218817 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4952b1-ccfb5b59f-4r2rj"] Nov 03 01:44:27.113942 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:44:27.113899 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-4952b1-ccfb5b59f-4r2rj" event={"ID":"6faa241e-9510-45af-9e59-e3aa35524217","Type":"ContainerStarted","Data":"9f70104848ce70798044dcf6a223e6d420153c7a815ba11c6ae036766afdebf8"} Nov 03 01:44:27.114138 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:44:27.113993 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-4952b1-ccfb5b59f-4r2rj" event={"ID":"6faa241e-9510-45af-9e59-e3aa35524217","Type":"ContainerStarted","Data":"782b07eaa4d43eda95fe4dda1c206b299adc6cf4e803fb90fc2f11f611ae4dea"} Nov 03 01:44:27.114138 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:44:27.114018 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-4952b1-ccfb5b59f-4r2rj" Nov 03 01:44:27.238455 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:44:27.238359 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-4952b1-ccfb5b59f-4r2rj" podStartSLOduration=2.238341242 podStartE2EDuration="2.238341242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:44:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:44:27.235289223 +0000 UTC m=+1549.264771522" watchObservedRunningTime="2026-11-03 01:44:27.238341242 +0000 UTC m=+1549.267823537" Nov 03 01:44:28.976247 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:44:28.976200 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:44:31.953132 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:44:31.953084 5437 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-2b9f22c1-7cd5c774dc-xbtwr" podUID="18277241-9019-4b65-86b7-7084af0b742d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:44:33.625454 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:44:33.625407 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:44:33.964555 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:44:33.964447 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:44:35.445786 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:44:35.445759 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-2b9f22c1-7cd5c774dc-xbtwr" Nov 03 01:44:35.560839 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:44:35.560805 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18277241-9019-4b65-86b7-7084af0b742d-kube-api-access-nwrks\" (UniqueName: \"kubernetes.io/projected/18277241-9019-4b65-86b7-7084af0b742d-kube-api-access-nwrks\") pod \"18277241-9019-4b65-86b7-7084af0b742d\" (UID: \"18277241-9019-4b65-86b7-7084af0b742d\") " Nov 03 01:44:35.561179 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:44:35.561153 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18277241-9019-4b65-86b7-7084af0b742d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18277241-9019-4b65-86b7-7084af0b742d-plugins-config-volume\") pod \"18277241-9019-4b65-86b7-7084af0b742d\" (UID: \"18277241-9019-4b65-86b7-7084af0b742d\") " Nov 03 01:44:35.561698 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:44:35.561673 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18277241-9019-4b65-86b7-7084af0b742d-plugins-config-volume" pod "18277241-9019-4b65-86b7-7084af0b742d" (UID: "18277241-9019-4b65-86b7-7084af0b742d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:44:35.563569 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:44:35.563532 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18277241-9019-4b65-86b7-7084af0b742d-kube-api-access-nwrks" pod "18277241-9019-4b65-86b7-7084af0b742d" (UID: "18277241-9019-4b65-86b7-7084af0b742d"). InnerVolumeSpecName "kube-api-access-nwrks". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:44:35.662626 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:44:35.662593 5437 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18277241-9019-4b65-86b7-7084af0b742d-plugins-config-volume\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:44:35.662626 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:44:35.662623 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwrks\" (UniqueName: \"kubernetes.io/projected/18277241-9019-4b65-86b7-7084af0b742d-kube-api-access-nwrks\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:44:36.181586 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:44:36.181549 5437 generic.go:356] "Generic (PLEG): container finished" podID="18277241-9019-4b65-86b7-7084af0b742d" containerID="ea4323a5a1d4f1e24a8d1e202b285fb2f3b47673dbb910e48df50b1e9ec430cd" exitCode=0 Nov 03 01:44:36.181792 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:44:36.181602 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-2b9f22c1-7cd5c774dc-xbtwr" event={"ID":"18277241-9019-4b65-86b7-7084af0b742d","Type":"ContainerDied","Data":"ea4323a5a1d4f1e24a8d1e202b285fb2f3b47673dbb910e48df50b1e9ec430cd"} Nov 03 01:44:36.181792 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:44:36.181650 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-2b9f22c1-7cd5c774dc-xbtwr" event={"ID":"18277241-9019-4b65-86b7-7084af0b742d","Type":"ContainerDied","Data":"eea6add51c3925f0e21b7a67e96a187d9fbb283bb82a312fddd42c4ef735ad95"} Nov 03 01:44:36.181792 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:44:36.181680 5437 scope.go:122] "RemoveContainer" containerID="ea4323a5a1d4f1e24a8d1e202b285fb2f3b47673dbb910e48df50b1e9ec430cd" Nov 03 01:44:36.181968 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:44:36.181851 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-2b9f22c1-7cd5c774dc-xbtwr" Nov 03 01:44:36.192054 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:44:36.192033 5437 scope.go:122] "RemoveContainer" containerID="ea4323a5a1d4f1e24a8d1e202b285fb2f3b47673dbb910e48df50b1e9ec430cd" Nov 03 01:44:36.192453 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:44:36.192416 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea4323a5a1d4f1e24a8d1e202b285fb2f3b47673dbb910e48df50b1e9ec430cd\": container with ID starting with ea4323a5a1d4f1e24a8d1e202b285fb2f3b47673dbb910e48df50b1e9ec430cd not found: ID does not exist" containerID="ea4323a5a1d4f1e24a8d1e202b285fb2f3b47673dbb910e48df50b1e9ec430cd" Nov 03 01:44:36.192557 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:44:36.192457 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea4323a5a1d4f1e24a8d1e202b285fb2f3b47673dbb910e48df50b1e9ec430cd"} err="failed to get container status \"ea4323a5a1d4f1e24a8d1e202b285fb2f3b47673dbb910e48df50b1e9ec430cd\": rpc error: code = NotFound desc = could not find container \"ea4323a5a1d4f1e24a8d1e202b285fb2f3b47673dbb910e48df50b1e9ec430cd\": container with ID starting with ea4323a5a1d4f1e24a8d1e202b285fb2f3b47673dbb910e48df50b1e9ec430cd not found: ID does not exist" Nov 03 01:44:36.358302 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:44:36.358220 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2b9f22c1-7cd5c774dc-xbtwr"] Nov 03 01:44:36.368066 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:44:36.368035 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2b9f22c1-7cd5c774dc-xbtwr"] Nov 03 01:44:38.111644 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:44:38.111610 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18277241-9019-4b65-86b7-7084af0b742d" path="/var/lib/kubelet/pods/18277241-9019-4b65-86b7-7084af0b742d/volumes" Nov 03 01:44:38.124277 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:44:38.124247 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-4952b1-ccfb5b59f-4r2rj" Nov 03 01:44:38.416262 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:44:38.416218 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f4e704-7dbf54c85-hvn9h"] Nov 03 01:44:38.417090 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:44:38.417066 5437 cpu_manager.go:414] "containerMap: removing container" podUID="18277241-9019-4b65-86b7-7084af0b742d" containerName="payload-pre-processing" Nov 03 01:44:38.417090 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:44:38.417089 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18277241-9019-4b65-86b7-7084af0b742d" containerName="payload-pre-processing" Nov 03 01:44:38.417288 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:44:38.417273 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="18277241-9019-4b65-86b7-7084af0b742d" containerName="payload-pre-processing" Nov 03 01:44:38.422227 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:44:38.422196 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f4e704-7dbf54c85-hvn9h" Nov 03 01:44:38.474755 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:44:38.474716 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f4e704-7dbf54c85-hvn9h"] Nov 03 01:44:38.477114 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:44:38.477083 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-f4e704-serving-cert\"" Nov 03 01:44:38.502567 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:44:38.502526 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87a89600-532d-4c7a-99f4-a9d41a58bfc5-maas-api-tls\") pod \"maas-api-e2e-api-a-f4e704-7dbf54c85-hvn9h\" (UID: \"87a89600-532d-4c7a-99f4-a9d41a58bfc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f4e704-7dbf54c85-hvn9h" Nov 03 01:44:38.502765 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:44:38.502576 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhfmr\" (UniqueName: \"kubernetes.io/projected/87a89600-532d-4c7a-99f4-a9d41a58bfc5-kube-api-access-dhfmr\") pod \"maas-api-e2e-api-a-f4e704-7dbf54c85-hvn9h\" (UID: \"87a89600-532d-4c7a-99f4-a9d41a58bfc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f4e704-7dbf54c85-hvn9h" Nov 03 01:44:38.502765 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:44:38.502744 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87a89600-532d-4c7a-99f4-a9d41a58bfc5-openshift-service-ca\") pod \"maas-api-e2e-api-a-f4e704-7dbf54c85-hvn9h\" (UID: \"87a89600-532d-4c7a-99f4-a9d41a58bfc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f4e704-7dbf54c85-hvn9h" Nov 03 01:44:38.502894 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:44:38.502867 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87a89600-532d-4c7a-99f4-a9d41a58bfc5-metrics-certs\") pod \"maas-api-e2e-api-a-f4e704-7dbf54c85-hvn9h\" (UID: \"87a89600-532d-4c7a-99f4-a9d41a58bfc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f4e704-7dbf54c85-hvn9h" Nov 03 01:44:38.603820 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:44:38.603775 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87a89600-532d-4c7a-99f4-a9d41a58bfc5-metrics-certs\") pod \"maas-api-e2e-api-a-f4e704-7dbf54c85-hvn9h\" (UID: \"87a89600-532d-4c7a-99f4-a9d41a58bfc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f4e704-7dbf54c85-hvn9h" Nov 03 01:44:38.604031 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:44:38.603917 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87a89600-532d-4c7a-99f4-a9d41a58bfc5-maas-api-tls\") pod \"maas-api-e2e-api-a-f4e704-7dbf54c85-hvn9h\" (UID: \"87a89600-532d-4c7a-99f4-a9d41a58bfc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f4e704-7dbf54c85-hvn9h" Nov 03 01:44:38.604031 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:44:38.603945 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhfmr\" (UniqueName: \"kubernetes.io/projected/87a89600-532d-4c7a-99f4-a9d41a58bfc5-kube-api-access-dhfmr\") pod \"maas-api-e2e-api-a-f4e704-7dbf54c85-hvn9h\" (UID: \"87a89600-532d-4c7a-99f4-a9d41a58bfc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f4e704-7dbf54c85-hvn9h" Nov 03 01:44:38.604157 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:44:38.604066 5437 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-f4e704-serving-cert: secret "maas-api-e2e-api-a-f4e704-serving-cert" not found Nov 03 01:44:38.604157 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:44:38.604081 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87a89600-532d-4c7a-99f4-a9d41a58bfc5-openshift-service-ca\") pod \"maas-api-e2e-api-a-f4e704-7dbf54c85-hvn9h\" (UID: \"87a89600-532d-4c7a-99f4-a9d41a58bfc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f4e704-7dbf54c85-hvn9h" Nov 03 01:44:38.604157 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:44:38.604146 5437 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87a89600-532d-4c7a-99f4-a9d41a58bfc5-maas-api-tls podName:87a89600-532d-4c7a-99f4-a9d41a58bfc5 nodeName:}" failed. No retries permitted until 2026-11-03 01:44:39.104124803 +0000 UTC m=+1561.133607089 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/87a89600-532d-4c7a-99f4-a9d41a58bfc5-maas-api-tls") pod "maas-api-e2e-api-a-f4e704-7dbf54c85-hvn9h" (UID: "87a89600-532d-4c7a-99f4-a9d41a58bfc5") : secret "maas-api-e2e-api-a-f4e704-serving-cert" not found Nov 03 01:44:38.604719 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:44:38.604695 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87a89600-532d-4c7a-99f4-a9d41a58bfc5-openshift-service-ca\") pod \"maas-api-e2e-api-a-f4e704-7dbf54c85-hvn9h\" (UID: \"87a89600-532d-4c7a-99f4-a9d41a58bfc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f4e704-7dbf54c85-hvn9h" Nov 03 01:44:38.606455 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:44:38.606432 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87a89600-532d-4c7a-99f4-a9d41a58bfc5-metrics-certs\") pod \"maas-api-e2e-api-a-f4e704-7dbf54c85-hvn9h\" (UID: \"87a89600-532d-4c7a-99f4-a9d41a58bfc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f4e704-7dbf54c85-hvn9h" Nov 03 01:44:38.669541 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:44:38.669430 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:44:38.678292 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:44:38.678254 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhfmr\" (UniqueName: \"kubernetes.io/projected/87a89600-532d-4c7a-99f4-a9d41a58bfc5-kube-api-access-dhfmr\") pod \"maas-api-e2e-api-a-f4e704-7dbf54c85-hvn9h\" (UID: \"87a89600-532d-4c7a-99f4-a9d41a58bfc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f4e704-7dbf54c85-hvn9h" Nov 03 01:44:38.880538 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:44:38.880499 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:44:39.082595 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:44:39.082551 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:44:39.112231 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:44:39.112189 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87a89600-532d-4c7a-99f4-a9d41a58bfc5-maas-api-tls\") pod \"maas-api-e2e-api-a-f4e704-7dbf54c85-hvn9h\" (UID: \"87a89600-532d-4c7a-99f4-a9d41a58bfc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f4e704-7dbf54c85-hvn9h" Nov 03 01:44:39.114787 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:44:39.114747 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87a89600-532d-4c7a-99f4-a9d41a58bfc5-maas-api-tls\") pod \"maas-api-e2e-api-a-f4e704-7dbf54c85-hvn9h\" (UID: \"87a89600-532d-4c7a-99f4-a9d41a58bfc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f4e704-7dbf54c85-hvn9h" Nov 03 01:44:39.338912 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:44:39.338818 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f4e704-7dbf54c85-hvn9h" Nov 03 01:44:39.715297 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:44:39.715266 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87a89600_532d_4c7a_99f4_a9d41a58bfc5.slice/crio-c776aa502e6e0d6e01d22d2d43a4aca94ba10061cc6f751d565593ee01c89c30 WatchSource:0}: Error finding container c776aa502e6e0d6e01d22d2d43a4aca94ba10061cc6f751d565593ee01c89c30: Status 404 returned error can't find the container with id c776aa502e6e0d6e01d22d2d43a4aca94ba10061cc6f751d565593ee01c89c30 Nov 03 01:44:39.718082 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:44:39.718054 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f4e704-7dbf54c85-hvn9h"] Nov 03 01:44:40.210277 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:44:40.210238 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f4e704-7dbf54c85-hvn9h" event={"ID":"87a89600-532d-4c7a-99f4-a9d41a58bfc5","Type":"ContainerStarted","Data":"c776aa502e6e0d6e01d22d2d43a4aca94ba10061cc6f751d565593ee01c89c30"} Nov 03 01:44:41.219222 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:44:41.219183 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f4e704-7dbf54c85-hvn9h" event={"ID":"87a89600-532d-4c7a-99f4-a9d41a58bfc5","Type":"ContainerStarted","Data":"dc6c078c7976366764c3a13b55f1d21b84fe511433d672a9fe39f5a119af8537"} Nov 03 01:44:41.219774 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:44:41.219257 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f4e704-7dbf54c85-hvn9h" Nov 03 01:44:41.341707 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:44:41.341653 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f4e704-7dbf54c85-hvn9h" podStartSLOduration=2.423355032 podStartE2EDuration="3.341637405s" totalImagesPullingTime="918.282373ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:44:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:44:41.340396242 +0000 UTC m=+1563.369878540" watchObservedRunningTime="2026-11-03 01:44:41.341637405 +0000 UTC m=+1563.371119738" Nov 03 01:44:43.538650 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:44:43.538601 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:44:43.943664 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:44:43.943621 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:44:44.039731 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:44:44.039695 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 01:44:44.267193 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:44:44.267101 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:44:44.492333 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:44:44.492288 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:44:45.355037 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:44:45.355000 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:44:45.579837 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:44:45.579804 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:44:47.232261 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:44:47.232225 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f4e704-7dbf54c85-hvn9h" Nov 03 01:44:49.139217 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:44:49.139172 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:44:49.566686 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:44:49.566643 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:44:49.825010 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:44:49.824917 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:44:52.715730 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:44:52.715694 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:44:52.920530 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:44:52.920493 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:44:53.043382 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:44:53.043275 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-118857-5db4c99879-84r9l"] Nov 03 01:44:53.048878 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:44:53.048851 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-118857-5db4c99879-84r9l" Nov 03 01:44:53.111451 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:44:53.111410 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-118857-5db4c99879-84r9l"] Nov 03 01:44:53.121870 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:44:53.121721 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crvt4\" (UniqueName: \"kubernetes.io/projected/a3547027-5292-4db3-8d5e-fa7e0f792d8f-kube-api-access-crvt4\") pod \"payload-pre-processing-e2e-api-b-118857-5db4c99879-84r9l\" (UID: \"a3547027-5292-4db3-8d5e-fa7e0f792d8f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-118857-5db4c99879-84r9l" Nov 03 01:44:53.121870 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:44:53.121840 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3547027-5292-4db3-8d5e-fa7e0f792d8f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-118857-5db4c99879-84r9l\" (UID: \"a3547027-5292-4db3-8d5e-fa7e0f792d8f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-118857-5db4c99879-84r9l" Nov 03 01:44:53.157787 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:44:53.157729 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-118857\"" Nov 03 01:44:53.158066 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:44:53.158043 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-118857-dockercfg-jhcc9\"" Nov 03 01:44:53.232391 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:44:53.231565 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crvt4\" (UniqueName: \"kubernetes.io/projected/a3547027-5292-4db3-8d5e-fa7e0f792d8f-kube-api-access-crvt4\") pod \"payload-pre-processing-e2e-api-b-118857-5db4c99879-84r9l\" (UID: \"a3547027-5292-4db3-8d5e-fa7e0f792d8f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-118857-5db4c99879-84r9l" Nov 03 01:44:53.232391 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:44:53.231688 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3547027-5292-4db3-8d5e-fa7e0f792d8f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-118857-5db4c99879-84r9l\" (UID: \"a3547027-5292-4db3-8d5e-fa7e0f792d8f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-118857-5db4c99879-84r9l" Nov 03 01:44:53.236386 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:44:53.233519 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3547027-5292-4db3-8d5e-fa7e0f792d8f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-118857-5db4c99879-84r9l\" (UID: \"a3547027-5292-4db3-8d5e-fa7e0f792d8f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-118857-5db4c99879-84r9l" Nov 03 01:44:53.305437 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:44:53.305356 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crvt4\" (UniqueName: \"kubernetes.io/projected/a3547027-5292-4db3-8d5e-fa7e0f792d8f-kube-api-access-crvt4\") pod \"payload-pre-processing-e2e-api-b-118857-5db4c99879-84r9l\" (UID: \"a3547027-5292-4db3-8d5e-fa7e0f792d8f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-118857-5db4c99879-84r9l" Nov 03 01:44:53.364296 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:44:53.364254 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-118857-5db4c99879-84r9l" Nov 03 01:44:53.757645 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:44:53.756486 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-118857-5db4c99879-84r9l"] Nov 03 01:44:54.323460 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:44:54.323419 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-118857-5db4c99879-84r9l" event={"ID":"a3547027-5292-4db3-8d5e-fa7e0f792d8f","Type":"ContainerStarted","Data":"b3325a09dd9703ee04da42cf6131460c735280b9a971d12e55f659b916a3b139"} Nov 03 01:44:54.323681 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:44:54.323479 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-118857-5db4c99879-84r9l" event={"ID":"a3547027-5292-4db3-8d5e-fa7e0f792d8f","Type":"ContainerStarted","Data":"c6db455e3bd9fa9c15486cd418b92267b720a7b5f352a502e579d0781fbf9afc"} Nov 03 01:44:54.323681 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:44:54.323608 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-118857-5db4c99879-84r9l" Nov 03 01:44:54.365719 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:44:54.363143 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:44:54.447445 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:44:54.447348 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-118857-5db4c99879-84r9l" podStartSLOduration=2.447328297 podStartE2EDuration="2.447328297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:44:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:44:54.443703047 +0000 UTC m=+1576.473185343" watchObservedRunningTime="2026-11-03 01:44:54.447328297 +0000 UTC m=+1576.476810595" Nov 03 01:44:55.683166 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:44:55.683117 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:44:57.390254 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:44:57.390210 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:44:59.579472 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:44:59.579428 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:45:00.215739 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:00.215702 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-51bcf3-29894505-c8npt"] Nov 03 01:45:00.220789 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:00.220759 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-51bcf3-29894505-c8npt" Nov 03 01:45:00.249455 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:00.249409 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnhsl\" (UniqueName: \"kubernetes.io/projected/c3f8a3e9-ae5d-4029-8fb1-633693e63ac1-kube-api-access-lnhsl\") pod \"maas-api-key-cleanup-e2e-api-a-51bcf3-29894505-c8npt\" (UID: \"c3f8a3e9-ae5d-4029-8fb1-633693e63ac1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-51bcf3-29894505-c8npt" Nov 03 01:45:00.249757 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:00.249727 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3f8a3e9-ae5d-4029-8fb1-633693e63ac1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-51bcf3-29894505-c8npt\" (UID: \"c3f8a3e9-ae5d-4029-8fb1-633693e63ac1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-51bcf3-29894505-c8npt" Nov 03 01:45:00.292564 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:00.292521 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-51bcf3-29894505-c8npt"] Nov 03 01:45:00.354784 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:00.354751 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fgwsl\"" Nov 03 01:45:00.379804 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:00.379757 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3f8a3e9-ae5d-4029-8fb1-633693e63ac1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-51bcf3-29894505-c8npt\" (UID: \"c3f8a3e9-ae5d-4029-8fb1-633693e63ac1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-51bcf3-29894505-c8npt" Nov 03 01:45:00.381010 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:00.380844 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3f8a3e9-ae5d-4029-8fb1-633693e63ac1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-51bcf3-29894505-c8npt\" (UID: \"c3f8a3e9-ae5d-4029-8fb1-633693e63ac1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-51bcf3-29894505-c8npt" Nov 03 01:45:00.396386 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:00.395581 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnhsl\" (UniqueName: \"kubernetes.io/projected/c3f8a3e9-ae5d-4029-8fb1-633693e63ac1-kube-api-access-lnhsl\") pod \"maas-api-key-cleanup-e2e-api-a-51bcf3-29894505-c8npt\" (UID: \"c3f8a3e9-ae5d-4029-8fb1-633693e63ac1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-51bcf3-29894505-c8npt" Nov 03 01:45:00.475577 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:00.475490 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnhsl\" (UniqueName: \"kubernetes.io/projected/c3f8a3e9-ae5d-4029-8fb1-633693e63ac1-kube-api-access-lnhsl\") pod \"maas-api-key-cleanup-e2e-api-a-51bcf3-29894505-c8npt\" (UID: \"c3f8a3e9-ae5d-4029-8fb1-633693e63ac1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-51bcf3-29894505-c8npt" Nov 03 01:45:00.536084 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:00.536042 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-51bcf3-29894505-c8npt" Nov 03 01:45:00.919554 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:45:00.919515 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3f8a3e9_ae5d_4029_8fb1_633693e63ac1.slice/crio-b0ca55603fb446c04bc61d814cba00d85733278d50b308f4387058919b652a5a WatchSource:0}: Error finding container b0ca55603fb446c04bc61d814cba00d85733278d50b308f4387058919b652a5a: Status 404 returned error can't find the container with id b0ca55603fb446c04bc61d814cba00d85733278d50b308f4387058919b652a5a Nov 03 01:45:00.922865 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:00.922837 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-51bcf3-29894505-c8npt"] Nov 03 01:45:01.386390 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:01.382722 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-51bcf3-29894505-c8npt" event={"ID":"c3f8a3e9-ae5d-4029-8fb1-633693e63ac1","Type":"ContainerStarted","Data":"b0ca55603fb446c04bc61d814cba00d85733278d50b308f4387058919b652a5a"} Nov 03 01:45:02.391754 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:02.391716 5437 generic.go:356] "Generic (PLEG): container finished" podID="c3f8a3e9-ae5d-4029-8fb1-633693e63ac1" containerID="9bf19013e4628e7919813e96171f88eacaf2e4399be35845bfbdb7be15bb5242" exitCode=60 Nov 03 01:45:02.392245 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:02.391788 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-51bcf3-29894505-c8npt" event={"ID":"c3f8a3e9-ae5d-4029-8fb1-633693e63ac1","Type":"ContainerDied","Data":"9bf19013e4628e7919813e96171f88eacaf2e4399be35845bfbdb7be15bb5242"} Nov 03 01:45:02.392245 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:02.392198 5437 scope.go:122] "RemoveContainer" containerID="9bf19013e4628e7919813e96171f88eacaf2e4399be35845bfbdb7be15bb5242" Nov 03 01:45:03.188994 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:03.188944 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:45:03.391652 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:03.391558 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:45:03.402651 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:03.402617 5437 generic.go:356] "Generic (PLEG): container finished" podID="c3f8a3e9-ae5d-4029-8fb1-633693e63ac1" containerID="0489cd0885571402a80ab532587c0adab8642f4971d307846c8ec23c13e09ce2" exitCode=60 Nov 03 01:45:03.403137 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:03.402696 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-51bcf3-29894505-c8npt" event={"ID":"c3f8a3e9-ae5d-4029-8fb1-633693e63ac1","Type":"ContainerDied","Data":"0489cd0885571402a80ab532587c0adab8642f4971d307846c8ec23c13e09ce2"} Nov 03 01:45:03.403137 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:03.402759 5437 scope.go:122] "RemoveContainer" containerID="9bf19013e4628e7919813e96171f88eacaf2e4399be35845bfbdb7be15bb5242" Nov 03 01:45:03.403137 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:03.403099 5437 scope.go:122] "RemoveContainer" containerID="0489cd0885571402a80ab532587c0adab8642f4971d307846c8ec23c13e09ce2" Nov 03 01:45:03.403539 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:45:03.403512 5437 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-51bcf3-29894505-c8npt_odh-ai-gateway-infra(c3f8a3e9-ae5d-4029-8fb1-633693e63ac1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-51bcf3-29894505-c8npt" podUID="c3f8a3e9-ae5d-4029-8fb1-633693e63ac1" Nov 03 01:45:04.412395 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:04.412316 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:45:04.421407 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:04.421354 5437 scope.go:122] "RemoveContainer" containerID="0489cd0885571402a80ab532587c0adab8642f4971d307846c8ec23c13e09ce2" Nov 03 01:45:04.421819 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:45:04.421790 5437 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-51bcf3-29894505-c8npt_odh-ai-gateway-infra(c3f8a3e9-ae5d-4029-8fb1-633693e63ac1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-51bcf3-29894505-c8npt" podUID="c3f8a3e9-ae5d-4029-8fb1-633693e63ac1" Nov 03 01:45:05.071030 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:05.070988 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:45:05.334556 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:05.334477 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-118857-5db4c99879-84r9l" Nov 03 01:45:09.969150 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:09.969112 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:45:10.188685 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:10.188650 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:45:12.247040 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:12.247002 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-cvlqf"] Nov 03 01:45:12.252507 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:12.252476 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-cvlqf" Nov 03 01:45:12.306314 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:12.306278 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-f4e704-openshift-default-dockercfg-nt8mh\"" Nov 03 01:45:12.311520 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:12.311486 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea981b45-743a-4e06-83c9-519fbda3d7d8-istiod-ca-cert\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-cvlqf\" (UID: \"ea981b45-743a-4e06-83c9-519fbda3d7d8\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-cvlqf" Nov 03 01:45:12.311697 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:12.311622 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2q6f7\" (UniqueName: \"kubernetes.io/projected/ea981b45-743a-4e06-83c9-519fbda3d7d8-kube-api-access-2q6f7\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-cvlqf\" (UID: \"ea981b45-743a-4e06-83c9-519fbda3d7d8\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-cvlqf" Nov 03 01:45:12.311697 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:12.311659 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea981b45-743a-4e06-83c9-519fbda3d7d8-istio-envoy\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-cvlqf\" (UID: \"ea981b45-743a-4e06-83c9-519fbda3d7d8\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-cvlqf" Nov 03 01:45:12.311803 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:12.311708 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea981b45-743a-4e06-83c9-519fbda3d7d8-istio-podinfo\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-cvlqf\" (UID: \"ea981b45-743a-4e06-83c9-519fbda3d7d8\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-cvlqf" Nov 03 01:45:12.311850 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:12.311800 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea981b45-743a-4e06-83c9-519fbda3d7d8-istio-token\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-cvlqf\" (UID: \"ea981b45-743a-4e06-83c9-519fbda3d7d8\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-cvlqf" Nov 03 01:45:12.311891 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:12.311863 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea981b45-743a-4e06-83c9-519fbda3d7d8-workload-certs\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-cvlqf\" (UID: \"ea981b45-743a-4e06-83c9-519fbda3d7d8\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-cvlqf" Nov 03 01:45:12.311943 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:12.311927 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea981b45-743a-4e06-83c9-519fbda3d7d8-credential-socket\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-cvlqf\" (UID: \"ea981b45-743a-4e06-83c9-519fbda3d7d8\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-cvlqf" Nov 03 01:45:12.312011 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:12.311986 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea981b45-743a-4e06-83c9-519fbda3d7d8-istio-data\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-cvlqf\" (UID: \"ea981b45-743a-4e06-83c9-519fbda3d7d8\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-cvlqf" Nov 03 01:45:12.312143 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:12.312128 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea981b45-743a-4e06-83c9-519fbda3d7d8-workload-socket\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-cvlqf\" (UID: \"ea981b45-743a-4e06-83c9-519fbda3d7d8\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-cvlqf" Nov 03 01:45:12.378748 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:12.378709 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-cvlqf"] Nov 03 01:45:12.425444 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:12.413576 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea981b45-743a-4e06-83c9-519fbda3d7d8-workload-socket\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-cvlqf\" (UID: \"ea981b45-743a-4e06-83c9-519fbda3d7d8\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-cvlqf" Nov 03 01:45:12.425444 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:12.413709 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea981b45-743a-4e06-83c9-519fbda3d7d8-istiod-ca-cert\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-cvlqf\" (UID: \"ea981b45-743a-4e06-83c9-519fbda3d7d8\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-cvlqf" Nov 03 01:45:12.425444 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:12.413805 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2q6f7\" (UniqueName: \"kubernetes.io/projected/ea981b45-743a-4e06-83c9-519fbda3d7d8-kube-api-access-2q6f7\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-cvlqf\" (UID: \"ea981b45-743a-4e06-83c9-519fbda3d7d8\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-cvlqf" Nov 03 01:45:12.425444 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:12.413847 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea981b45-743a-4e06-83c9-519fbda3d7d8-istio-envoy\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-cvlqf\" (UID: \"ea981b45-743a-4e06-83c9-519fbda3d7d8\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-cvlqf" Nov 03 01:45:12.425444 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:12.413911 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea981b45-743a-4e06-83c9-519fbda3d7d8-istio-podinfo\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-cvlqf\" (UID: \"ea981b45-743a-4e06-83c9-519fbda3d7d8\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-cvlqf" Nov 03 01:45:12.425444 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:12.413954 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea981b45-743a-4e06-83c9-519fbda3d7d8-istio-token\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-cvlqf\" (UID: \"ea981b45-743a-4e06-83c9-519fbda3d7d8\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-cvlqf" Nov 03 01:45:12.425444 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:12.414010 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea981b45-743a-4e06-83c9-519fbda3d7d8-workload-certs\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-cvlqf\" (UID: \"ea981b45-743a-4e06-83c9-519fbda3d7d8\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-cvlqf" Nov 03 01:45:12.425444 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:12.414106 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea981b45-743a-4e06-83c9-519fbda3d7d8-credential-socket\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-cvlqf\" (UID: \"ea981b45-743a-4e06-83c9-519fbda3d7d8\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-cvlqf" Nov 03 01:45:12.425444 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:12.414132 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea981b45-743a-4e06-83c9-519fbda3d7d8-istio-data\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-cvlqf\" (UID: \"ea981b45-743a-4e06-83c9-519fbda3d7d8\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-cvlqf" Nov 03 01:45:12.425444 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:12.414639 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea981b45-743a-4e06-83c9-519fbda3d7d8-istio-data\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-cvlqf\" (UID: \"ea981b45-743a-4e06-83c9-519fbda3d7d8\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-cvlqf" Nov 03 01:45:12.425444 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:12.414865 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea981b45-743a-4e06-83c9-519fbda3d7d8-workload-socket\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-cvlqf\" (UID: \"ea981b45-743a-4e06-83c9-519fbda3d7d8\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-cvlqf" Nov 03 01:45:12.425444 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:12.415484 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea981b45-743a-4e06-83c9-519fbda3d7d8-istiod-ca-cert\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-cvlqf\" (UID: \"ea981b45-743a-4e06-83c9-519fbda3d7d8\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-cvlqf" Nov 03 01:45:12.425444 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:12.416322 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea981b45-743a-4e06-83c9-519fbda3d7d8-workload-certs\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-cvlqf\" (UID: \"ea981b45-743a-4e06-83c9-519fbda3d7d8\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-cvlqf" Nov 03 01:45:12.425444 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:12.416584 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea981b45-743a-4e06-83c9-519fbda3d7d8-credential-socket\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-cvlqf\" (UID: \"ea981b45-743a-4e06-83c9-519fbda3d7d8\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-cvlqf" Nov 03 01:45:12.425444 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:12.419458 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea981b45-743a-4e06-83c9-519fbda3d7d8-istio-podinfo\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-cvlqf\" (UID: \"ea981b45-743a-4e06-83c9-519fbda3d7d8\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-cvlqf" Nov 03 01:45:12.426060 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:12.425861 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea981b45-743a-4e06-83c9-519fbda3d7d8-istio-envoy\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-cvlqf\" (UID: \"ea981b45-743a-4e06-83c9-519fbda3d7d8\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-cvlqf" Nov 03 01:45:12.479255 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:12.479209 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2q6f7\" (UniqueName: \"kubernetes.io/projected/ea981b45-743a-4e06-83c9-519fbda3d7d8-kube-api-access-2q6f7\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-cvlqf\" (UID: \"ea981b45-743a-4e06-83c9-519fbda3d7d8\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-cvlqf" Nov 03 01:45:12.479767 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:12.479742 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea981b45-743a-4e06-83c9-519fbda3d7d8-istio-token\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-cvlqf\" (UID: \"ea981b45-743a-4e06-83c9-519fbda3d7d8\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-cvlqf" Nov 03 01:45:12.568111 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:12.568077 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-cvlqf" Nov 03 01:45:12.993322 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:45:12.993287 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea981b45_743a_4e06_83c9_519fbda3d7d8.slice/crio-161527535e07fbfb420154a31d4303afb152c5d9682e416103f052258ab1dff5 WatchSource:0}: Error finding container 161527535e07fbfb420154a31d4303afb152c5d9682e416103f052258ab1dff5: Status 404 returned error can't find the container with id 161527535e07fbfb420154a31d4303afb152c5d9682e416103f052258ab1dff5 Nov 03 01:45:12.994462 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:12.994429 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-cvlqf"] Nov 03 01:45:12.996251 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:12.996211 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 01:45:12.996404 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:12.996291 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 01:45:12.996404 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:12.996342 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 01:45:13.492086 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:13.491982 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-cvlqf" event={"ID":"ea981b45-743a-4e06-83c9-519fbda3d7d8","Type":"ContainerStarted","Data":"2901cf672fe080b882e623aea3d6c30aacf3f3ed81c901e8186e1f49c085b123"} Nov 03 01:45:13.492086 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:13.492037 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-cvlqf" event={"ID":"ea981b45-743a-4e06-83c9-519fbda3d7d8","Type":"ContainerStarted","Data":"161527535e07fbfb420154a31d4303afb152c5d9682e416103f052258ab1dff5"} Nov 03 01:45:13.568870 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:13.568839 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-cvlqf" Nov 03 01:45:13.654640 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:13.654580 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-cvlqf" podStartSLOduration=1.654563631 podStartE2EDuration="1.654563631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:45:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:45:13.653496352 +0000 UTC m=+1595.682978649" watchObservedRunningTime="2026-11-03 01:45:13.654563631 +0000 UTC m=+1595.684045928" Nov 03 01:45:14.052392 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:45:14.052343 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 01:45:14.570945 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:14.570904 5437 patch_prober.go:29] interesting pod/e2e-api-a-f4e704-openshift-default-576d67d949-cvlqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Nov 03 01:45:14.571378 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:14.570964 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-cvlqf" podUID="ea981b45-743a-4e06-83c9-519fbda3d7d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Nov 03 01:45:14.745141 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:14.745107 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:45:15.569693 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:15.569647 5437 patch_prober.go:29] interesting pod/e2e-api-a-f4e704-openshift-default-576d67d949-cvlqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Nov 03 01:45:15.569883 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:15.569722 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-cvlqf" podUID="ea981b45-743a-4e06-83c9-519fbda3d7d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Nov 03 01:45:16.082082 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:16.082048 5437 scope.go:122] "RemoveContainer" containerID="0489cd0885571402a80ab532587c0adab8642f4971d307846c8ec23c13e09ce2" Nov 03 01:45:16.277096 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:16.277056 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:45:16.568669 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:16.568641 5437 patch_prober.go:29] interesting pod/e2e-api-a-f4e704-openshift-default-576d67d949-cvlqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Nov 03 01:45:16.568822 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:16.568691 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-cvlqf" podUID="ea981b45-743a-4e06-83c9-519fbda3d7d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Nov 03 01:45:16.584183 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:16.584146 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:45:16.792792 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:16.792701 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:45:17.551813 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:17.551776 5437 generic.go:356] "Generic (PLEG): container finished" podID="c3f8a3e9-ae5d-4029-8fb1-633693e63ac1" containerID="5d56c9dc89b0bd1d5233630689c1941a71a3dc685237da112f783f204f94efae" exitCode=60 Nov 03 01:45:17.552266 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:17.551851 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-51bcf3-29894505-c8npt" event={"ID":"c3f8a3e9-ae5d-4029-8fb1-633693e63ac1","Type":"ContainerDied","Data":"5d56c9dc89b0bd1d5233630689c1941a71a3dc685237da112f783f204f94efae"} Nov 03 01:45:17.552266 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:17.551971 5437 scope.go:122] "RemoveContainer" containerID="0489cd0885571402a80ab532587c0adab8642f4971d307846c8ec23c13e09ce2" Nov 03 01:45:17.552526 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:17.552509 5437 scope.go:122] "RemoveContainer" containerID="5d56c9dc89b0bd1d5233630689c1941a71a3dc685237da112f783f204f94efae" Nov 03 01:45:17.552918 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:45:17.552891 5437 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-51bcf3-29894505-c8npt_odh-ai-gateway-infra(c3f8a3e9-ae5d-4029-8fb1-633693e63ac1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-51bcf3-29894505-c8npt" podUID="c3f8a3e9-ae5d-4029-8fb1-633693e63ac1" Nov 03 01:45:17.569018 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:17.568995 5437 patch_prober.go:29] interesting pod/e2e-api-a-f4e704-openshift-default-576d67d949-cvlqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Nov 03 01:45:17.569122 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:17.569055 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-cvlqf" podUID="ea981b45-743a-4e06-83c9-519fbda3d7d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Nov 03 01:45:18.569048 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:18.569014 5437 patch_prober.go:29] interesting pod/e2e-api-a-f4e704-openshift-default-576d67d949-cvlqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Nov 03 01:45:18.569581 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:18.569084 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-cvlqf" podUID="ea981b45-743a-4e06-83c9-519fbda3d7d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Nov 03 01:45:18.682584 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:18.682543 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-51bcf3-29894505-c8npt"] Nov 03 01:45:18.902136 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:18.902102 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-51bcf3-29894505-c8npt" Nov 03 01:45:19.054831 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:19.054796 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3f8a3e9-ae5d-4029-8fb1-633693e63ac1-kube-api-access-lnhsl\" (UniqueName: \"kubernetes.io/projected/c3f8a3e9-ae5d-4029-8fb1-633693e63ac1-kube-api-access-lnhsl\") pod \"c3f8a3e9-ae5d-4029-8fb1-633693e63ac1\" (UID: \"c3f8a3e9-ae5d-4029-8fb1-633693e63ac1\") " Nov 03 01:45:19.055043 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:19.054919 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3f8a3e9-ae5d-4029-8fb1-633693e63ac1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3f8a3e9-ae5d-4029-8fb1-633693e63ac1-maas-api-service-ca\") pod \"c3f8a3e9-ae5d-4029-8fb1-633693e63ac1\" (UID: \"c3f8a3e9-ae5d-4029-8fb1-633693e63ac1\") " Nov 03 01:45:19.055945 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:19.055909 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3f8a3e9-ae5d-4029-8fb1-633693e63ac1-maas-api-service-ca" pod "c3f8a3e9-ae5d-4029-8fb1-633693e63ac1" (UID: "c3f8a3e9-ae5d-4029-8fb1-633693e63ac1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:45:19.060434 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:19.058084 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3f8a3e9-ae5d-4029-8fb1-633693e63ac1-kube-api-access-lnhsl" pod "c3f8a3e9-ae5d-4029-8fb1-633693e63ac1" (UID: "c3f8a3e9-ae5d-4029-8fb1-633693e63ac1"). InnerVolumeSpecName "kube-api-access-lnhsl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:45:19.157121 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:19.157032 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnhsl\" (UniqueName: \"kubernetes.io/projected/c3f8a3e9-ae5d-4029-8fb1-633693e63ac1-kube-api-access-lnhsl\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:45:19.157121 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:19.157062 5437 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3f8a3e9-ae5d-4029-8fb1-633693e63ac1-maas-api-service-ca\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:45:19.567816 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:19.567786 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-51bcf3-29894505-c8npt" Nov 03 01:45:19.568025 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:19.567823 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-51bcf3-29894505-c8npt" event={"ID":"c3f8a3e9-ae5d-4029-8fb1-633693e63ac1","Type":"ContainerDied","Data":"b0ca55603fb446c04bc61d814cba00d85733278d50b308f4387058919b652a5a"} Nov 03 01:45:19.568025 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:19.567899 5437 scope.go:122] "RemoveContainer" containerID="5d56c9dc89b0bd1d5233630689c1941a71a3dc685237da112f783f204f94efae" Nov 03 01:45:19.573679 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:19.573652 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-cvlqf" Nov 03 01:45:19.574117 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:19.574098 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-cvlqf" Nov 03 01:45:19.574905 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:19.574884 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-cvlqf" Nov 03 01:45:19.863441 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:19.863303 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-51bcf3-29894505-c8npt"] Nov 03 01:45:19.869705 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:19.869669 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-51bcf3-29894505-c8npt"] Nov 03 01:45:20.089454 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:20.089418 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3f8a3e9-ae5d-4029-8fb1-633693e63ac1" path="/var/lib/kubelet/pods/c3f8a3e9-ae5d-4029-8fb1-633693e63ac1/volumes" Nov 03 01:45:21.582337 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:21.582290 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:45:21.786080 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:21.786041 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:45:26.206167 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:26.199898 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:45:26.460789 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:26.460693 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:45:28.052968 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:28.052928 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:45:30.129866 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:30.129834 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:45:30.388229 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:30.388138 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:45:32.337578 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:32.337529 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:45:32.593941 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:32.593841 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:45:33.363245 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:33.363203 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:45:33.779348 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:33.779247 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:45:33.986896 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:33.986859 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:45:35.338392 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:35.331695 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:45:35.872514 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:35.872476 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:45:36.629855 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:36.629811 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:45:38.423518 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:38.420626 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:45:41.559986 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:41.559942 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-g5ggm"] Nov 03 01:45:41.560471 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:41.560189 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-g5ggm" podUID="1ab5aefa-9401-4a6d-9168-fff8422de5b3" containerName="istio-proxy" containerID="cri-o://d8ac94044857ee28411f971121094fc30c570e3c75a574878566c5931896bdfa" gracePeriod=30 Nov 03 01:45:42.826018 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:42.825971 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:45:43.108874 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:43.108761 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:45:43.358796 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:43.358742 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:45:43.766639 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:43.766598 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:45:44.050400 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:45:44.050340 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 01:45:44.823094 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:44.823058 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:45:46.713720 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:46.713680 5437 patch_prober.go:29] interesting pod/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-g5ggm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 03 01:45:46.714136 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:46.713757 5437 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-g5ggm" podUID="1ab5aefa-9401-4a6d-9168-fff8422de5b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 03 01:45:46.780008 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:46.779961 5437 generic.go:356] "Generic (PLEG): container finished" podID="1ab5aefa-9401-4a6d-9168-fff8422de5b3" containerID="d8ac94044857ee28411f971121094fc30c570e3c75a574878566c5931896bdfa" exitCode=0 Nov 03 01:45:46.780208 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:46.780038 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-g5ggm" event={"ID":"1ab5aefa-9401-4a6d-9168-fff8422de5b3","Type":"ContainerDied","Data":"d8ac94044857ee28411f971121094fc30c570e3c75a574878566c5931896bdfa"} Nov 03 01:45:46.840502 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:46.840397 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-g5ggm" Nov 03 01:45:46.859810 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:46.859770 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1ab5aefa-9401-4a6d-9168-fff8422de5b3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ab5aefa-9401-4a6d-9168-fff8422de5b3-istio-podinfo\") pod \"1ab5aefa-9401-4a6d-9168-fff8422de5b3\" (UID: \"1ab5aefa-9401-4a6d-9168-fff8422de5b3\") " Nov 03 01:45:46.860018 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:46.859837 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ab5aefa-9401-4a6d-9168-fff8422de5b3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ab5aefa-9401-4a6d-9168-fff8422de5b3-workload-certs\") pod \"1ab5aefa-9401-4a6d-9168-fff8422de5b3\" (UID: \"1ab5aefa-9401-4a6d-9168-fff8422de5b3\") " Nov 03 01:45:46.860018 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:46.859868 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ab5aefa-9401-4a6d-9168-fff8422de5b3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ab5aefa-9401-4a6d-9168-fff8422de5b3-istio-data\") pod \"1ab5aefa-9401-4a6d-9168-fff8422de5b3\" (UID: \"1ab5aefa-9401-4a6d-9168-fff8422de5b3\") " Nov 03 01:45:46.860018 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:46.859910 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ab5aefa-9401-4a6d-9168-fff8422de5b3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ab5aefa-9401-4a6d-9168-fff8422de5b3-istiod-ca-cert\") pod \"1ab5aefa-9401-4a6d-9168-fff8422de5b3\" (UID: \"1ab5aefa-9401-4a6d-9168-fff8422de5b3\") " Nov 03 01:45:46.860018 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:46.859937 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ab5aefa-9401-4a6d-9168-fff8422de5b3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ab5aefa-9401-4a6d-9168-fff8422de5b3-workload-socket\") pod \"1ab5aefa-9401-4a6d-9168-fff8422de5b3\" (UID: \"1ab5aefa-9401-4a6d-9168-fff8422de5b3\") " Nov 03 01:45:46.860238 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:46.860072 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ab5aefa-9401-4a6d-9168-fff8422de5b3-istio-token\" (UniqueName: \"kubernetes.io/projected/1ab5aefa-9401-4a6d-9168-fff8422de5b3-istio-token\") pod \"1ab5aefa-9401-4a6d-9168-fff8422de5b3\" (UID: \"1ab5aefa-9401-4a6d-9168-fff8422de5b3\") " Nov 03 01:45:46.860238 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:46.860127 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ab5aefa-9401-4a6d-9168-fff8422de5b3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ab5aefa-9401-4a6d-9168-fff8422de5b3-istio-envoy\") pod \"1ab5aefa-9401-4a6d-9168-fff8422de5b3\" (UID: \"1ab5aefa-9401-4a6d-9168-fff8422de5b3\") " Nov 03 01:45:46.860238 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:46.860187 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ab5aefa-9401-4a6d-9168-fff8422de5b3-kube-api-access-6qgf9\" (UniqueName: \"kubernetes.io/projected/1ab5aefa-9401-4a6d-9168-fff8422de5b3-kube-api-access-6qgf9\") pod \"1ab5aefa-9401-4a6d-9168-fff8422de5b3\" (UID: \"1ab5aefa-9401-4a6d-9168-fff8422de5b3\") " Nov 03 01:45:46.860509 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:46.860233 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ab5aefa-9401-4a6d-9168-fff8422de5b3-workload-socket" pod "1ab5aefa-9401-4a6d-9168-fff8422de5b3" (UID: "1ab5aefa-9401-4a6d-9168-fff8422de5b3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:45:46.860509 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:46.860250 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ab5aefa-9401-4a6d-9168-fff8422de5b3-workload-certs" pod "1ab5aefa-9401-4a6d-9168-fff8422de5b3" (UID: "1ab5aefa-9401-4a6d-9168-fff8422de5b3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:45:46.860509 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:46.860258 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ab5aefa-9401-4a6d-9168-fff8422de5b3-istio-data" pod "1ab5aefa-9401-4a6d-9168-fff8422de5b3" (UID: "1ab5aefa-9401-4a6d-9168-fff8422de5b3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:45:46.860509 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:46.860314 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ab5aefa-9401-4a6d-9168-fff8422de5b3-istiod-ca-cert" pod "1ab5aefa-9401-4a6d-9168-fff8422de5b3" (UID: "1ab5aefa-9401-4a6d-9168-fff8422de5b3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:45:46.860509 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:46.860476 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ab5aefa-9401-4a6d-9168-fff8422de5b3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ab5aefa-9401-4a6d-9168-fff8422de5b3-credential-socket\") pod \"1ab5aefa-9401-4a6d-9168-fff8422de5b3\" (UID: \"1ab5aefa-9401-4a6d-9168-fff8422de5b3\") " Nov 03 01:45:46.860782 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:46.860757 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ab5aefa-9401-4a6d-9168-fff8422de5b3-credential-socket" pod "1ab5aefa-9401-4a6d-9168-fff8422de5b3" (UID: "1ab5aefa-9401-4a6d-9168-fff8422de5b3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:45:46.861342 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:46.861315 5437 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ab5aefa-9401-4a6d-9168-fff8422de5b3-workload-certs\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:45:46.861493 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:46.861346 5437 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ab5aefa-9401-4a6d-9168-fff8422de5b3-istio-data\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:45:46.861493 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:46.861379 5437 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ab5aefa-9401-4a6d-9168-fff8422de5b3-istiod-ca-cert\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:45:46.861493 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:46.861393 5437 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ab5aefa-9401-4a6d-9168-fff8422de5b3-workload-socket\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:45:46.861493 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:46.861409 5437 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ab5aefa-9401-4a6d-9168-fff8422de5b3-credential-socket\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:45:46.862447 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:46.862415 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1ab5aefa-9401-4a6d-9168-fff8422de5b3-istio-podinfo" pod "1ab5aefa-9401-4a6d-9168-fff8422de5b3" (UID: "1ab5aefa-9401-4a6d-9168-fff8422de5b3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:45:46.862547 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:46.862425 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ab5aefa-9401-4a6d-9168-fff8422de5b3-istio-token" pod "1ab5aefa-9401-4a6d-9168-fff8422de5b3" (UID: "1ab5aefa-9401-4a6d-9168-fff8422de5b3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:45:46.862714 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:46.862689 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ab5aefa-9401-4a6d-9168-fff8422de5b3-istio-envoy" pod "1ab5aefa-9401-4a6d-9168-fff8422de5b3" (UID: "1ab5aefa-9401-4a6d-9168-fff8422de5b3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:45:46.862811 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:46.862755 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ab5aefa-9401-4a6d-9168-fff8422de5b3-kube-api-access-6qgf9" pod "1ab5aefa-9401-4a6d-9168-fff8422de5b3" (UID: "1ab5aefa-9401-4a6d-9168-fff8422de5b3"). InnerVolumeSpecName "kube-api-access-6qgf9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:45:46.962716 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:46.962657 5437 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ab5aefa-9401-4a6d-9168-fff8422de5b3-istio-podinfo\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:45:46.962716 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:46.962705 5437 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ab5aefa-9401-4a6d-9168-fff8422de5b3-istio-token\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:45:46.962716 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:46.962715 5437 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ab5aefa-9401-4a6d-9168-fff8422de5b3-istio-envoy\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:45:46.962716 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:46.962724 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qgf9\" (UniqueName: \"kubernetes.io/projected/1ab5aefa-9401-4a6d-9168-fff8422de5b3-kube-api-access-6qgf9\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:45:47.788507 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:47.788468 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-g5ggm" event={"ID":"1ab5aefa-9401-4a6d-9168-fff8422de5b3","Type":"ContainerDied","Data":"cf91bc23675ef02df18f80fdc041f9236279fd74782273ce9f6acdaca197d891"} Nov 03 01:45:47.788507 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:47.788509 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-g5ggm" Nov 03 01:45:47.788976 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:47.788535 5437 scope.go:122] "RemoveContainer" containerID="d8ac94044857ee28411f971121094fc30c570e3c75a574878566c5931896bdfa" Nov 03 01:45:48.016497 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:48.016458 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-g5ggm"] Nov 03 01:45:48.020358 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:48.020325 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-g5ggm"] Nov 03 01:45:48.092965 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:48.092932 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ab5aefa-9401-4a6d-9168-fff8422de5b3" path="/var/lib/kubelet/pods/1ab5aefa-9401-4a6d-9168-fff8422de5b3/volumes" Nov 03 01:45:48.623526 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:48.623493 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:45:48.829288 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:48.829235 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:45:49.052644 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:49.052600 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:45:49.269383 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:49.269340 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:45:49.492131 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:49.492029 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:45:49.699447 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:49.699406 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:45:50.272206 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:50.269936 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:45:50.673201 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:50.673169 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:45:53.342664 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:53.342618 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:45:54.632555 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:54.632506 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:45:54.840885 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:54.840830 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:45:55.214464 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:55.214359 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:45:56.337245 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:56.337208 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:45:56.549432 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:56.549391 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:45:57.607247 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:57.607210 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-pm59c"] Nov 03 01:45:57.607997 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:57.607974 5437 cpu_manager.go:414] "containerMap: removing container" podUID="c3f8a3e9-ae5d-4029-8fb1-633693e63ac1" containerName="cleanup" Nov 03 01:45:57.607997 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:57.607996 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3f8a3e9-ae5d-4029-8fb1-633693e63ac1" containerName="cleanup" Nov 03 01:45:57.608177 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:57.608031 5437 cpu_manager.go:414] "containerMap: removing container" podUID="c3f8a3e9-ae5d-4029-8fb1-633693e63ac1" containerName="cleanup" Nov 03 01:45:57.608177 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:57.608037 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3f8a3e9-ae5d-4029-8fb1-633693e63ac1" containerName="cleanup" Nov 03 01:45:57.608177 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:57.608048 5437 cpu_manager.go:414] "containerMap: removing container" podUID="1ab5aefa-9401-4a6d-9168-fff8422de5b3" containerName="istio-proxy" Nov 03 01:45:57.608177 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:57.608054 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ab5aefa-9401-4a6d-9168-fff8422de5b3" containerName="istio-proxy" Nov 03 01:45:57.608302 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:57.608185 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3f8a3e9-ae5d-4029-8fb1-633693e63ac1" containerName="cleanup" Nov 03 01:45:57.608302 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:57.608196 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3f8a3e9-ae5d-4029-8fb1-633693e63ac1" containerName="cleanup" Nov 03 01:45:57.608302 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:57.608205 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ab5aefa-9401-4a6d-9168-fff8422de5b3" containerName="istio-proxy" Nov 03 01:45:57.611726 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:57.611707 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-pm59c" Nov 03 01:45:57.721055 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:57.721018 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-pm59c"] Nov 03 01:45:57.742626 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:57.742581 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pfh9\" (UniqueName: \"kubernetes.io/projected/5b441095-b0c0-4bbe-89af-5cf30727a4a7-kube-api-access-6pfh9\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-pm59c\" (UID: \"5b441095-b0c0-4bbe-89af-5cf30727a4a7\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-pm59c" Nov 03 01:45:57.742811 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:57.742714 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b441095-b0c0-4bbe-89af-5cf30727a4a7-workload-certs\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-pm59c\" (UID: \"5b441095-b0c0-4bbe-89af-5cf30727a4a7\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-pm59c" Nov 03 01:45:57.742811 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:57.742765 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b441095-b0c0-4bbe-89af-5cf30727a4a7-istio-token\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-pm59c\" (UID: \"5b441095-b0c0-4bbe-89af-5cf30727a4a7\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-pm59c" Nov 03 01:45:57.742911 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:57.742843 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b441095-b0c0-4bbe-89af-5cf30727a4a7-istio-data\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-pm59c\" (UID: \"5b441095-b0c0-4bbe-89af-5cf30727a4a7\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-pm59c" Nov 03 01:45:57.742911 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:57.742887 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b441095-b0c0-4bbe-89af-5cf30727a4a7-credential-socket\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-pm59c\" (UID: \"5b441095-b0c0-4bbe-89af-5cf30727a4a7\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-pm59c" Nov 03 01:45:57.743011 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:57.742979 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b441095-b0c0-4bbe-89af-5cf30727a4a7-istio-podinfo\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-pm59c\" (UID: \"5b441095-b0c0-4bbe-89af-5cf30727a4a7\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-pm59c" Nov 03 01:45:57.743160 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:57.743138 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b441095-b0c0-4bbe-89af-5cf30727a4a7-istiod-ca-cert\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-pm59c\" (UID: \"5b441095-b0c0-4bbe-89af-5cf30727a4a7\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-pm59c" Nov 03 01:45:57.743231 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:57.743218 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b441095-b0c0-4bbe-89af-5cf30727a4a7-istio-envoy\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-pm59c\" (UID: \"5b441095-b0c0-4bbe-89af-5cf30727a4a7\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-pm59c" Nov 03 01:45:57.743271 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:57.743255 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b441095-b0c0-4bbe-89af-5cf30727a4a7-workload-socket\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-pm59c\" (UID: \"5b441095-b0c0-4bbe-89af-5cf30727a4a7\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-pm59c" Nov 03 01:45:57.766997 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:57.766959 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:45:57.854844 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:57.851967 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b441095-b0c0-4bbe-89af-5cf30727a4a7-istiod-ca-cert\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-pm59c\" (UID: \"5b441095-b0c0-4bbe-89af-5cf30727a4a7\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-pm59c" Nov 03 01:45:57.854844 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:57.852051 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b441095-b0c0-4bbe-89af-5cf30727a4a7-istio-envoy\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-pm59c\" (UID: \"5b441095-b0c0-4bbe-89af-5cf30727a4a7\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-pm59c" Nov 03 01:45:57.859293 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:57.852096 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b441095-b0c0-4bbe-89af-5cf30727a4a7-workload-socket\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-pm59c\" (UID: \"5b441095-b0c0-4bbe-89af-5cf30727a4a7\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-pm59c" Nov 03 01:45:57.859293 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:57.858657 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pfh9\" (UniqueName: \"kubernetes.io/projected/5b441095-b0c0-4bbe-89af-5cf30727a4a7-kube-api-access-6pfh9\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-pm59c\" (UID: \"5b441095-b0c0-4bbe-89af-5cf30727a4a7\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-pm59c" Nov 03 01:45:57.859293 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:57.858739 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b441095-b0c0-4bbe-89af-5cf30727a4a7-workload-certs\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-pm59c\" (UID: \"5b441095-b0c0-4bbe-89af-5cf30727a4a7\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-pm59c" Nov 03 01:45:57.859293 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:57.858769 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b441095-b0c0-4bbe-89af-5cf30727a4a7-istio-token\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-pm59c\" (UID: \"5b441095-b0c0-4bbe-89af-5cf30727a4a7\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-pm59c" Nov 03 01:45:57.859293 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:57.858803 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b441095-b0c0-4bbe-89af-5cf30727a4a7-istio-data\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-pm59c\" (UID: \"5b441095-b0c0-4bbe-89af-5cf30727a4a7\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-pm59c" Nov 03 01:45:57.859293 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:57.858848 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b441095-b0c0-4bbe-89af-5cf30727a4a7-credential-socket\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-pm59c\" (UID: \"5b441095-b0c0-4bbe-89af-5cf30727a4a7\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-pm59c" Nov 03 01:45:57.859293 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:57.858904 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b441095-b0c0-4bbe-89af-5cf30727a4a7-istio-podinfo\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-pm59c\" (UID: \"5b441095-b0c0-4bbe-89af-5cf30727a4a7\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-pm59c" Nov 03 01:45:57.863308 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:57.860222 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b441095-b0c0-4bbe-89af-5cf30727a4a7-istiod-ca-cert\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-pm59c\" (UID: \"5b441095-b0c0-4bbe-89af-5cf30727a4a7\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-pm59c" Nov 03 01:45:57.864862 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:57.864828 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b441095-b0c0-4bbe-89af-5cf30727a4a7-workload-certs\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-pm59c\" (UID: \"5b441095-b0c0-4bbe-89af-5cf30727a4a7\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-pm59c" Nov 03 01:45:57.865317 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:57.865254 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b441095-b0c0-4bbe-89af-5cf30727a4a7-workload-socket\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-pm59c\" (UID: \"5b441095-b0c0-4bbe-89af-5cf30727a4a7\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-pm59c" Nov 03 01:45:57.866629 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:57.865890 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b441095-b0c0-4bbe-89af-5cf30727a4a7-istio-data\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-pm59c\" (UID: \"5b441095-b0c0-4bbe-89af-5cf30727a4a7\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-pm59c" Nov 03 01:45:57.884328 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:57.884230 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b441095-b0c0-4bbe-89af-5cf30727a4a7-istio-envoy\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-pm59c\" (UID: \"5b441095-b0c0-4bbe-89af-5cf30727a4a7\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-pm59c" Nov 03 01:45:57.884822 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:57.884563 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b441095-b0c0-4bbe-89af-5cf30727a4a7-istio-podinfo\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-pm59c\" (UID: \"5b441095-b0c0-4bbe-89af-5cf30727a4a7\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-pm59c" Nov 03 01:45:57.884822 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:57.884697 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b441095-b0c0-4bbe-89af-5cf30727a4a7-credential-socket\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-pm59c\" (UID: \"5b441095-b0c0-4bbe-89af-5cf30727a4a7\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-pm59c" Nov 03 01:45:57.931760 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:57.931725 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pfh9\" (UniqueName: \"kubernetes.io/projected/5b441095-b0c0-4bbe-89af-5cf30727a4a7-kube-api-access-6pfh9\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-pm59c\" (UID: \"5b441095-b0c0-4bbe-89af-5cf30727a4a7\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-pm59c" Nov 03 01:45:57.931935 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:57.931725 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b441095-b0c0-4bbe-89af-5cf30727a4a7-istio-token\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-pm59c\" (UID: \"5b441095-b0c0-4bbe-89af-5cf30727a4a7\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-pm59c" Nov 03 01:45:57.968485 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:57.968331 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:45:58.175164 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:58.174981 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:45:58.253889 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:58.253839 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-pm59c" Nov 03 01:45:58.288008 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:58.284839 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-ltbdx"] Nov 03 01:45:58.288008 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:58.285210 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-ltbdx" podUID="18705810-dd2e-41bc-886a-f8ec91844c8f" containerName="istio-proxy" containerID="cri-o://828b378979be082d469c55917aa81a591296ba654f2dd4d6ebf988c99b0368cf" gracePeriod=30 Nov 03 01:45:58.288008 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:58.286098 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-x7znt"] Nov 03 01:45:58.288008 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:58.286291 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-x7znt" podUID="de0ef434-0294-4dba-8112-317298a2acb5" containerName="istio-proxy" containerID="cri-o://4d91470adf4671f55a8a7b9bc5f3866698157cd49eb0b5d735340df9460edcde" gracePeriod=30 Nov 03 01:45:58.568481 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:58.568435 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:45:58.764095 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:45:58.764063 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b441095_b0c0_4bbe_89af_5cf30727a4a7.slice/crio-c28f38c17c4b176b8a3b2a79c539e930a00b23e1b2801e32253cbbdc71d19c42 WatchSource:0}: Error finding container c28f38c17c4b176b8a3b2a79c539e930a00b23e1b2801e32253cbbdc71d19c42: Status 404 returned error can't find the container with id c28f38c17c4b176b8a3b2a79c539e930a00b23e1b2801e32253cbbdc71d19c42 Nov 03 01:45:58.767314 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:58.767270 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 01:45:58.767460 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:58.767329 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-pm59c"] Nov 03 01:45:58.767460 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:58.767353 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 01:45:58.767460 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:58.767434 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 01:45:58.898932 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:58.898892 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-pm59c" event={"ID":"5b441095-b0c0-4bbe-89af-5cf30727a4a7","Type":"ContainerStarted","Data":"689cceff9761fcc394d4b801a557ed706bb8bd78ceb053ef08185016c9898043"} Nov 03 01:45:58.899146 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:58.898963 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-pm59c" event={"ID":"5b441095-b0c0-4bbe-89af-5cf30727a4a7","Type":"ContainerStarted","Data":"c28f38c17c4b176b8a3b2a79c539e930a00b23e1b2801e32253cbbdc71d19c42"} Nov 03 01:45:59.024269 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:59.024149 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-pm59c" podStartSLOduration=2.024133825 podStartE2EDuration="2.024133825s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:45:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:45:59.022327763 +0000 UTC m=+1641.051810058" watchObservedRunningTime="2026-11-03 01:45:59.024133825 +0000 UTC m=+1641.053616119" Nov 03 01:45:59.255350 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:59.255300 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-pm59c" Nov 03 01:45:59.257331 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:59.257301 5437 patch_prober.go:29] interesting pod/e2e-api-a-f4e704-openshift-default-576d67d949-pm59c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Nov 03 01:45:59.257526 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:45:59.257360 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-pm59c" podUID="5b441095-b0c0-4bbe-89af-5cf30727a4a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Nov 03 01:46:00.256390 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:00.255961 5437 patch_prober.go:29] interesting pod/e2e-api-a-f4e704-openshift-default-576d67d949-pm59c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Nov 03 01:46:00.256390 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:00.256025 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-pm59c" podUID="5b441095-b0c0-4bbe-89af-5cf30727a4a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Nov 03 01:46:00.621391 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:00.618498 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:46:01.255642 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:01.255603 5437 patch_prober.go:29] interesting pod/e2e-api-a-f4e704-openshift-default-576d67d949-pm59c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Nov 03 01:46:01.255835 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:01.255663 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-pm59c" podUID="5b441095-b0c0-4bbe-89af-5cf30727a4a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Nov 03 01:46:02.255514 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:02.255473 5437 patch_prober.go:29] interesting pod/e2e-api-a-f4e704-openshift-default-576d67d949-pm59c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Nov 03 01:46:02.255930 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:02.255528 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-pm59c" podUID="5b441095-b0c0-4bbe-89af-5cf30727a4a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Nov 03 01:46:03.254696 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:03.254662 5437 patch_prober.go:29] interesting pod/e2e-api-a-f4e704-openshift-default-576d67d949-pm59c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Nov 03 01:46:03.254920 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:03.254718 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-pm59c" podUID="5b441095-b0c0-4bbe-89af-5cf30727a4a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Nov 03 01:46:03.444352 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:03.444312 5437 patch_prober.go:29] interesting pod/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-ltbdx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 03 01:46:03.444877 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:03.444415 5437 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-ltbdx" podUID="18705810-dd2e-41bc-886a-f8ec91844c8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 03 01:46:03.687560 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:03.687514 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-ltbdx" Nov 03 01:46:03.690932 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:03.690904 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-x7znt" Nov 03 01:46:03.786340 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:03.786302 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de0ef434-0294-4dba-8112-317298a2acb5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de0ef434-0294-4dba-8112-317298a2acb5-istio-envoy\") pod \"de0ef434-0294-4dba-8112-317298a2acb5\" (UID: \"de0ef434-0294-4dba-8112-317298a2acb5\") " Nov 03 01:46:03.786520 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:03.786464 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/18705810-dd2e-41bc-886a-f8ec91844c8f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18705810-dd2e-41bc-886a-f8ec91844c8f-istio-podinfo\") pod \"18705810-dd2e-41bc-886a-f8ec91844c8f\" (UID: \"18705810-dd2e-41bc-886a-f8ec91844c8f\") " Nov 03 01:46:03.786520 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:03.786499 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18705810-dd2e-41bc-886a-f8ec91844c8f-istio-token\" (UniqueName: \"kubernetes.io/projected/18705810-dd2e-41bc-886a-f8ec91844c8f-istio-token\") pod \"18705810-dd2e-41bc-886a-f8ec91844c8f\" (UID: \"18705810-dd2e-41bc-886a-f8ec91844c8f\") " Nov 03 01:46:03.786654 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:03.786530 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/de0ef434-0294-4dba-8112-317298a2acb5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de0ef434-0294-4dba-8112-317298a2acb5-istio-podinfo\") pod \"de0ef434-0294-4dba-8112-317298a2acb5\" (UID: \"de0ef434-0294-4dba-8112-317298a2acb5\") " Nov 03 01:46:03.786654 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:03.786571 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de0ef434-0294-4dba-8112-317298a2acb5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de0ef434-0294-4dba-8112-317298a2acb5-istiod-ca-cert\") pod \"de0ef434-0294-4dba-8112-317298a2acb5\" (UID: \"de0ef434-0294-4dba-8112-317298a2acb5\") " Nov 03 01:46:03.786875 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:03.786847 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18705810-dd2e-41bc-886a-f8ec91844c8f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18705810-dd2e-41bc-886a-f8ec91844c8f-istio-data\") pod \"18705810-dd2e-41bc-886a-f8ec91844c8f\" (UID: \"18705810-dd2e-41bc-886a-f8ec91844c8f\") " Nov 03 01:46:03.786958 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:03.786883 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de0ef434-0294-4dba-8112-317298a2acb5-istiod-ca-cert" pod "de0ef434-0294-4dba-8112-317298a2acb5" (UID: "de0ef434-0294-4dba-8112-317298a2acb5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:46:03.786958 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:03.786896 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18705810-dd2e-41bc-886a-f8ec91844c8f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18705810-dd2e-41bc-886a-f8ec91844c8f-istio-envoy\") pod \"18705810-dd2e-41bc-886a-f8ec91844c8f\" (UID: \"18705810-dd2e-41bc-886a-f8ec91844c8f\") " Nov 03 01:46:03.786958 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:03.786944 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de0ef434-0294-4dba-8112-317298a2acb5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de0ef434-0294-4dba-8112-317298a2acb5-credential-socket\") pod \"de0ef434-0294-4dba-8112-317298a2acb5\" (UID: \"de0ef434-0294-4dba-8112-317298a2acb5\") " Nov 03 01:46:03.787123 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:03.787063 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de0ef434-0294-4dba-8112-317298a2acb5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de0ef434-0294-4dba-8112-317298a2acb5-istio-data\") pod \"de0ef434-0294-4dba-8112-317298a2acb5\" (UID: \"de0ef434-0294-4dba-8112-317298a2acb5\") " Nov 03 01:46:03.787183 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:03.787128 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18705810-dd2e-41bc-886a-f8ec91844c8f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18705810-dd2e-41bc-886a-f8ec91844c8f-credential-socket\") pod \"18705810-dd2e-41bc-886a-f8ec91844c8f\" (UID: \"18705810-dd2e-41bc-886a-f8ec91844c8f\") " Nov 03 01:46:03.787183 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:03.787173 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de0ef434-0294-4dba-8112-317298a2acb5-kube-api-access-l6lvq\" (UniqueName: \"kubernetes.io/projected/de0ef434-0294-4dba-8112-317298a2acb5-kube-api-access-l6lvq\") pod \"de0ef434-0294-4dba-8112-317298a2acb5\" (UID: \"de0ef434-0294-4dba-8112-317298a2acb5\") " Nov 03 01:46:03.787274 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:03.787225 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de0ef434-0294-4dba-8112-317298a2acb5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de0ef434-0294-4dba-8112-317298a2acb5-workload-certs\") pod \"de0ef434-0294-4dba-8112-317298a2acb5\" (UID: \"de0ef434-0294-4dba-8112-317298a2acb5\") " Nov 03 01:46:03.787274 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:03.787267 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18705810-dd2e-41bc-886a-f8ec91844c8f-kube-api-access-zgqxs\" (UniqueName: \"kubernetes.io/projected/18705810-dd2e-41bc-886a-f8ec91844c8f-kube-api-access-zgqxs\") pod \"18705810-dd2e-41bc-886a-f8ec91844c8f\" (UID: \"18705810-dd2e-41bc-886a-f8ec91844c8f\") " Nov 03 01:46:03.787346 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:03.787270 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de0ef434-0294-4dba-8112-317298a2acb5-credential-socket" pod "de0ef434-0294-4dba-8112-317298a2acb5" (UID: "de0ef434-0294-4dba-8112-317298a2acb5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:46:03.787346 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:03.787298 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de0ef434-0294-4dba-8112-317298a2acb5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de0ef434-0294-4dba-8112-317298a2acb5-workload-socket\") pod \"de0ef434-0294-4dba-8112-317298a2acb5\" (UID: \"de0ef434-0294-4dba-8112-317298a2acb5\") " Nov 03 01:46:03.787506 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:03.787428 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18705810-dd2e-41bc-886a-f8ec91844c8f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18705810-dd2e-41bc-886a-f8ec91844c8f-workload-certs\") pod \"18705810-dd2e-41bc-886a-f8ec91844c8f\" (UID: \"18705810-dd2e-41bc-886a-f8ec91844c8f\") " Nov 03 01:46:03.787506 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:03.787429 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18705810-dd2e-41bc-886a-f8ec91844c8f-istio-data" pod "18705810-dd2e-41bc-886a-f8ec91844c8f" (UID: "18705810-dd2e-41bc-886a-f8ec91844c8f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:46:03.787620 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:03.787530 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18705810-dd2e-41bc-886a-f8ec91844c8f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18705810-dd2e-41bc-886a-f8ec91844c8f-istiod-ca-cert\") pod \"18705810-dd2e-41bc-886a-f8ec91844c8f\" (UID: \"18705810-dd2e-41bc-886a-f8ec91844c8f\") " Nov 03 01:46:03.787620 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:03.787570 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de0ef434-0294-4dba-8112-317298a2acb5-istio-token\" (UniqueName: \"kubernetes.io/projected/de0ef434-0294-4dba-8112-317298a2acb5-istio-token\") pod \"de0ef434-0294-4dba-8112-317298a2acb5\" (UID: \"de0ef434-0294-4dba-8112-317298a2acb5\") " Nov 03 01:46:03.787724 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:03.787621 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18705810-dd2e-41bc-886a-f8ec91844c8f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18705810-dd2e-41bc-886a-f8ec91844c8f-workload-socket\") pod \"18705810-dd2e-41bc-886a-f8ec91844c8f\" (UID: \"18705810-dd2e-41bc-886a-f8ec91844c8f\") " Nov 03 01:46:03.788720 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:03.788432 5437 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de0ef434-0294-4dba-8112-317298a2acb5-istiod-ca-cert\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:46:03.788720 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:03.788460 5437 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18705810-dd2e-41bc-886a-f8ec91844c8f-istio-data\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:46:03.788720 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:03.788476 5437 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de0ef434-0294-4dba-8112-317298a2acb5-credential-socket\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:46:03.789171 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:03.788999 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de0ef434-0294-4dba-8112-317298a2acb5-istio-envoy" pod "de0ef434-0294-4dba-8112-317298a2acb5" (UID: "de0ef434-0294-4dba-8112-317298a2acb5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:46:03.789171 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:03.789011 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18705810-dd2e-41bc-886a-f8ec91844c8f-workload-socket" pod "18705810-dd2e-41bc-886a-f8ec91844c8f" (UID: "18705810-dd2e-41bc-886a-f8ec91844c8f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:46:03.789345 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:03.789214 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/de0ef434-0294-4dba-8112-317298a2acb5-istio-podinfo" pod "de0ef434-0294-4dba-8112-317298a2acb5" (UID: "de0ef434-0294-4dba-8112-317298a2acb5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:46:03.789345 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:03.789221 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18705810-dd2e-41bc-886a-f8ec91844c8f-istio-token" pod "18705810-dd2e-41bc-886a-f8ec91844c8f" (UID: "18705810-dd2e-41bc-886a-f8ec91844c8f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:46:03.789345 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:03.789235 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de0ef434-0294-4dba-8112-317298a2acb5-workload-socket" pod "de0ef434-0294-4dba-8112-317298a2acb5" (UID: "de0ef434-0294-4dba-8112-317298a2acb5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:46:03.789345 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:03.789246 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de0ef434-0294-4dba-8112-317298a2acb5-istio-data" pod "de0ef434-0294-4dba-8112-317298a2acb5" (UID: "de0ef434-0294-4dba-8112-317298a2acb5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:46:03.789625 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:03.789428 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18705810-dd2e-41bc-886a-f8ec91844c8f-credential-socket" pod "18705810-dd2e-41bc-886a-f8ec91844c8f" (UID: "18705810-dd2e-41bc-886a-f8ec91844c8f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:46:03.789685 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:03.789623 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18705810-dd2e-41bc-886a-f8ec91844c8f-workload-certs" pod "18705810-dd2e-41bc-886a-f8ec91844c8f" (UID: "18705810-dd2e-41bc-886a-f8ec91844c8f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:46:03.789685 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:03.789596 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de0ef434-0294-4dba-8112-317298a2acb5-workload-certs" pod "de0ef434-0294-4dba-8112-317298a2acb5" (UID: "de0ef434-0294-4dba-8112-317298a2acb5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:46:03.789800 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:03.789712 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/18705810-dd2e-41bc-886a-f8ec91844c8f-istio-podinfo" pod "18705810-dd2e-41bc-886a-f8ec91844c8f" (UID: "18705810-dd2e-41bc-886a-f8ec91844c8f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:46:03.789800 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:03.789710 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18705810-dd2e-41bc-886a-f8ec91844c8f-istiod-ca-cert" pod "18705810-dd2e-41bc-886a-f8ec91844c8f" (UID: "18705810-dd2e-41bc-886a-f8ec91844c8f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:46:03.790031 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:03.790008 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de0ef434-0294-4dba-8112-317298a2acb5-kube-api-access-l6lvq" pod "de0ef434-0294-4dba-8112-317298a2acb5" (UID: "de0ef434-0294-4dba-8112-317298a2acb5"). InnerVolumeSpecName "kube-api-access-l6lvq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:46:03.790112 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:03.790069 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18705810-dd2e-41bc-886a-f8ec91844c8f-istio-envoy" pod "18705810-dd2e-41bc-886a-f8ec91844c8f" (UID: "18705810-dd2e-41bc-886a-f8ec91844c8f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:46:03.790534 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:03.790513 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18705810-dd2e-41bc-886a-f8ec91844c8f-kube-api-access-zgqxs" pod "18705810-dd2e-41bc-886a-f8ec91844c8f" (UID: "18705810-dd2e-41bc-886a-f8ec91844c8f"). InnerVolumeSpecName "kube-api-access-zgqxs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:46:03.791090 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:03.791075 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de0ef434-0294-4dba-8112-317298a2acb5-istio-token" pod "de0ef434-0294-4dba-8112-317298a2acb5" (UID: "de0ef434-0294-4dba-8112-317298a2acb5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:46:03.890112 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:03.890026 5437 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18705810-dd2e-41bc-886a-f8ec91844c8f-credential-socket\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:46:03.890112 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:03.890058 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6lvq\" (UniqueName: \"kubernetes.io/projected/de0ef434-0294-4dba-8112-317298a2acb5-kube-api-access-l6lvq\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:46:03.890112 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:03.890070 5437 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de0ef434-0294-4dba-8112-317298a2acb5-workload-certs\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:46:03.890112 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:03.890083 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgqxs\" (UniqueName: \"kubernetes.io/projected/18705810-dd2e-41bc-886a-f8ec91844c8f-kube-api-access-zgqxs\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:46:03.890112 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:03.890091 5437 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de0ef434-0294-4dba-8112-317298a2acb5-workload-socket\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:46:03.890112 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:03.890102 5437 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18705810-dd2e-41bc-886a-f8ec91844c8f-workload-certs\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:46:03.890112 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:03.890111 5437 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18705810-dd2e-41bc-886a-f8ec91844c8f-istiod-ca-cert\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:46:03.890112 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:03.890119 5437 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de0ef434-0294-4dba-8112-317298a2acb5-istio-token\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:46:03.890506 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:03.890128 5437 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18705810-dd2e-41bc-886a-f8ec91844c8f-workload-socket\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:46:03.890506 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:03.890139 5437 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de0ef434-0294-4dba-8112-317298a2acb5-istio-envoy\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:46:03.890506 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:03.890147 5437 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18705810-dd2e-41bc-886a-f8ec91844c8f-istio-podinfo\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:46:03.890506 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:03.890157 5437 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18705810-dd2e-41bc-886a-f8ec91844c8f-istio-token\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:46:03.890506 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:03.890166 5437 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de0ef434-0294-4dba-8112-317298a2acb5-istio-podinfo\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:46:03.890506 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:03.890176 5437 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18705810-dd2e-41bc-886a-f8ec91844c8f-istio-envoy\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:46:03.890506 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:03.890185 5437 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de0ef434-0294-4dba-8112-317298a2acb5-istio-data\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:46:03.955559 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:03.955517 5437 generic.go:356] "Generic (PLEG): container finished" podID="de0ef434-0294-4dba-8112-317298a2acb5" containerID="4d91470adf4671f55a8a7b9bc5f3866698157cd49eb0b5d735340df9460edcde" exitCode=0 Nov 03 01:46:03.955753 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:03.955739 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-x7znt" Nov 03 01:46:03.955813 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:03.955780 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-x7znt" event={"ID":"de0ef434-0294-4dba-8112-317298a2acb5","Type":"ContainerDied","Data":"4d91470adf4671f55a8a7b9bc5f3866698157cd49eb0b5d735340df9460edcde"} Nov 03 01:46:03.955914 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:03.955896 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-x7znt" event={"ID":"de0ef434-0294-4dba-8112-317298a2acb5","Type":"ContainerDied","Data":"d6b192f119ae80339c4c515f0bf3f65a96b36ade3618e7b98ebac9da7bdec974"} Nov 03 01:46:03.955951 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:03.955936 5437 scope.go:122] "RemoveContainer" containerID="4d91470adf4671f55a8a7b9bc5f3866698157cd49eb0b5d735340df9460edcde" Nov 03 01:46:03.957938 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:03.957906 5437 generic.go:356] "Generic (PLEG): container finished" podID="18705810-dd2e-41bc-886a-f8ec91844c8f" containerID="828b378979be082d469c55917aa81a591296ba654f2dd4d6ebf988c99b0368cf" exitCode=0 Nov 03 01:46:03.958104 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:03.957993 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-ltbdx" event={"ID":"18705810-dd2e-41bc-886a-f8ec91844c8f","Type":"ContainerDied","Data":"828b378979be082d469c55917aa81a591296ba654f2dd4d6ebf988c99b0368cf"} Nov 03 01:46:03.958104 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:03.958058 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-ltbdx" event={"ID":"18705810-dd2e-41bc-886a-f8ec91844c8f","Type":"ContainerDied","Data":"3b0123b7bd2dab840390c0cfa968728a427314a90fc0184eae76c84fc3ac6d7b"} Nov 03 01:46:03.958271 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:03.958251 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-ltbdx" Nov 03 01:46:03.985708 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:03.983905 5437 scope.go:122] "RemoveContainer" containerID="4d91470adf4671f55a8a7b9bc5f3866698157cd49eb0b5d735340df9460edcde" Nov 03 01:46:03.985708 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:46:03.984788 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d91470adf4671f55a8a7b9bc5f3866698157cd49eb0b5d735340df9460edcde\": container with ID starting with 4d91470adf4671f55a8a7b9bc5f3866698157cd49eb0b5d735340df9460edcde not found: ID does not exist" containerID="4d91470adf4671f55a8a7b9bc5f3866698157cd49eb0b5d735340df9460edcde" Nov 03 01:46:03.985708 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:03.984845 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d91470adf4671f55a8a7b9bc5f3866698157cd49eb0b5d735340df9460edcde"} err="failed to get container status \"4d91470adf4671f55a8a7b9bc5f3866698157cd49eb0b5d735340df9460edcde\": rpc error: code = NotFound desc = could not find container \"4d91470adf4671f55a8a7b9bc5f3866698157cd49eb0b5d735340df9460edcde\": container with ID starting with 4d91470adf4671f55a8a7b9bc5f3866698157cd49eb0b5d735340df9460edcde not found: ID does not exist" Nov 03 01:46:03.985708 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:03.984873 5437 scope.go:122] "RemoveContainer" containerID="828b378979be082d469c55917aa81a591296ba654f2dd4d6ebf988c99b0368cf" Nov 03 01:46:04.002121 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:04.002045 5437 scope.go:122] "RemoveContainer" containerID="828b378979be082d469c55917aa81a591296ba654f2dd4d6ebf988c99b0368cf" Nov 03 01:46:04.002591 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:46:04.002521 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"828b378979be082d469c55917aa81a591296ba654f2dd4d6ebf988c99b0368cf\": container with ID starting with 828b378979be082d469c55917aa81a591296ba654f2dd4d6ebf988c99b0368cf not found: ID does not exist" containerID="828b378979be082d469c55917aa81a591296ba654f2dd4d6ebf988c99b0368cf" Nov 03 01:46:04.002591 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:04.002559 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"828b378979be082d469c55917aa81a591296ba654f2dd4d6ebf988c99b0368cf"} err="failed to get container status \"828b378979be082d469c55917aa81a591296ba654f2dd4d6ebf988c99b0368cf\": rpc error: code = NotFound desc = could not find container \"828b378979be082d469c55917aa81a591296ba654f2dd4d6ebf988c99b0368cf\": container with ID starting with 828b378979be082d469c55917aa81a591296ba654f2dd4d6ebf988c99b0368cf not found: ID does not exist" Nov 03 01:46:04.153284 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:04.153144 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-x7znt"] Nov 03 01:46:04.157063 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:04.157036 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-x7znt"] Nov 03 01:46:04.161583 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:04.161552 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-gh4dm"] Nov 03 01:46:04.161845 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:04.161812 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-gh4dm" podUID="27d14b9a-3320-4e79-85b6-662d86877310" containerName="istio-proxy" containerID="cri-o://32940d908f0602e5c9546989b63eb6cd41bd35095d5bf1dcea6a5e8d190b5052" gracePeriod=30 Nov 03 01:46:04.167454 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:04.166833 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-m26wn"] Nov 03 01:46:04.167454 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:04.167068 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-m26wn" podUID="71058df3-a083-440b-b68a-5dd686509a66" containerName="istio-proxy" containerID="cri-o://21cf4f2e51142980b59d8e8f7f18a274a46eeb54c6305d7703ae7e12f38862dd" gracePeriod=30 Nov 03 01:46:04.256419 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:04.256283 5437 patch_prober.go:29] interesting pod/e2e-api-a-f4e704-openshift-default-576d67d949-pm59c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Nov 03 01:46:04.256419 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:04.256335 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-pm59c" podUID="5b441095-b0c0-4bbe-89af-5cf30727a4a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Nov 03 01:46:04.261939 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:04.261904 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-ltbdx"] Nov 03 01:46:04.265152 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:04.265116 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-ltbdx"] Nov 03 01:46:05.260989 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:05.260951 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-pm59c" Nov 03 01:46:05.261490 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:05.261472 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-pm59c" Nov 03 01:46:05.262159 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:05.262139 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-pm59c" Nov 03 01:46:05.835940 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:05.835901 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:46:06.101430 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:06.101319 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18705810-dd2e-41bc-886a-f8ec91844c8f" path="/var/lib/kubelet/pods/18705810-dd2e-41bc-886a-f8ec91844c8f/volumes" Nov 03 01:46:06.102541 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:06.102514 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de0ef434-0294-4dba-8112-317298a2acb5" path="/var/lib/kubelet/pods/de0ef434-0294-4dba-8112-317298a2acb5/volumes" Nov 03 01:46:09.472083 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:09.472055 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-gh4dm" Nov 03 01:46:09.500386 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:09.500342 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-m26wn" Nov 03 01:46:09.574004 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:09.573963 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71058df3-a083-440b-b68a-5dd686509a66-kube-api-access-2h22m\" (UniqueName: \"kubernetes.io/projected/71058df3-a083-440b-b68a-5dd686509a66-kube-api-access-2h22m\") pod \"71058df3-a083-440b-b68a-5dd686509a66\" (UID: \"71058df3-a083-440b-b68a-5dd686509a66\") " Nov 03 01:46:09.574004 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:09.574004 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71058df3-a083-440b-b68a-5dd686509a66-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71058df3-a083-440b-b68a-5dd686509a66-credential-socket\") pod \"71058df3-a083-440b-b68a-5dd686509a66\" (UID: \"71058df3-a083-440b-b68a-5dd686509a66\") " Nov 03 01:46:09.574231 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:09.574056 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/71058df3-a083-440b-b68a-5dd686509a66-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71058df3-a083-440b-b68a-5dd686509a66-istio-podinfo\") pod \"71058df3-a083-440b-b68a-5dd686509a66\" (UID: \"71058df3-a083-440b-b68a-5dd686509a66\") " Nov 03 01:46:09.574231 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:09.574082 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27d14b9a-3320-4e79-85b6-662d86877310-istio-token\" (UniqueName: \"kubernetes.io/projected/27d14b9a-3320-4e79-85b6-662d86877310-istio-token\") pod \"27d14b9a-3320-4e79-85b6-662d86877310\" (UID: \"27d14b9a-3320-4e79-85b6-662d86877310\") " Nov 03 01:46:09.574231 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:09.574146 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71058df3-a083-440b-b68a-5dd686509a66-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71058df3-a083-440b-b68a-5dd686509a66-workload-socket\") pod \"71058df3-a083-440b-b68a-5dd686509a66\" (UID: \"71058df3-a083-440b-b68a-5dd686509a66\") " Nov 03 01:46:09.574231 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:09.574181 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71058df3-a083-440b-b68a-5dd686509a66-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71058df3-a083-440b-b68a-5dd686509a66-workload-certs\") pod \"71058df3-a083-440b-b68a-5dd686509a66\" (UID: \"71058df3-a083-440b-b68a-5dd686509a66\") " Nov 03 01:46:09.574231 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:09.574209 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27d14b9a-3320-4e79-85b6-662d86877310-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27d14b9a-3320-4e79-85b6-662d86877310-workload-certs\") pod \"27d14b9a-3320-4e79-85b6-662d86877310\" (UID: \"27d14b9a-3320-4e79-85b6-662d86877310\") " Nov 03 01:46:09.574497 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:09.574468 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71058df3-a083-440b-b68a-5dd686509a66-credential-socket" pod "71058df3-a083-440b-b68a-5dd686509a66" (UID: "71058df3-a083-440b-b68a-5dd686509a66"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:46:09.574549 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:09.574491 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27d14b9a-3320-4e79-85b6-662d86877310-workload-certs" pod "27d14b9a-3320-4e79-85b6-662d86877310" (UID: "27d14b9a-3320-4e79-85b6-662d86877310"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:46:09.574549 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:09.574474 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71058df3-a083-440b-b68a-5dd686509a66-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71058df3-a083-440b-b68a-5dd686509a66-istio-envoy\") pod \"71058df3-a083-440b-b68a-5dd686509a66\" (UID: \"71058df3-a083-440b-b68a-5dd686509a66\") " Nov 03 01:46:09.574663 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:09.574566 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71058df3-a083-440b-b68a-5dd686509a66-istio-token\" (UniqueName: \"kubernetes.io/projected/71058df3-a083-440b-b68a-5dd686509a66-istio-token\") pod \"71058df3-a083-440b-b68a-5dd686509a66\" (UID: \"71058df3-a083-440b-b68a-5dd686509a66\") " Nov 03 01:46:09.574663 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:09.574644 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/27d14b9a-3320-4e79-85b6-662d86877310-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27d14b9a-3320-4e79-85b6-662d86877310-istio-podinfo\") pod \"27d14b9a-3320-4e79-85b6-662d86877310\" (UID: \"27d14b9a-3320-4e79-85b6-662d86877310\") " Nov 03 01:46:09.574758 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:09.574688 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71058df3-a083-440b-b68a-5dd686509a66-workload-certs" pod "71058df3-a083-440b-b68a-5dd686509a66" (UID: "71058df3-a083-440b-b68a-5dd686509a66"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:46:09.574758 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:09.574717 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71058df3-a083-440b-b68a-5dd686509a66-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71058df3-a083-440b-b68a-5dd686509a66-istiod-ca-cert\") pod \"71058df3-a083-440b-b68a-5dd686509a66\" (UID: \"71058df3-a083-440b-b68a-5dd686509a66\") " Nov 03 01:46:09.574871 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:09.574774 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27d14b9a-3320-4e79-85b6-662d86877310-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27d14b9a-3320-4e79-85b6-662d86877310-istiod-ca-cert\") pod \"27d14b9a-3320-4e79-85b6-662d86877310\" (UID: \"27d14b9a-3320-4e79-85b6-662d86877310\") " Nov 03 01:46:09.574925 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:09.574899 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27d14b9a-3320-4e79-85b6-662d86877310-kube-api-access-pjz8c\" (UniqueName: \"kubernetes.io/projected/27d14b9a-3320-4e79-85b6-662d86877310-kube-api-access-pjz8c\") pod \"27d14b9a-3320-4e79-85b6-662d86877310\" (UID: \"27d14b9a-3320-4e79-85b6-662d86877310\") " Nov 03 01:46:09.574979 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:09.574937 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27d14b9a-3320-4e79-85b6-662d86877310-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27d14b9a-3320-4e79-85b6-662d86877310-workload-socket\") pod \"27d14b9a-3320-4e79-85b6-662d86877310\" (UID: \"27d14b9a-3320-4e79-85b6-662d86877310\") " Nov 03 01:46:09.575042 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:09.574987 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71058df3-a083-440b-b68a-5dd686509a66-workload-socket" pod "71058df3-a083-440b-b68a-5dd686509a66" (UID: "71058df3-a083-440b-b68a-5dd686509a66"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:46:09.575042 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:09.575004 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27d14b9a-3320-4e79-85b6-662d86877310-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27d14b9a-3320-4e79-85b6-662d86877310-istio-envoy\") pod \"27d14b9a-3320-4e79-85b6-662d86877310\" (UID: \"27d14b9a-3320-4e79-85b6-662d86877310\") " Nov 03 01:46:09.575158 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:09.575088 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27d14b9a-3320-4e79-85b6-662d86877310-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27d14b9a-3320-4e79-85b6-662d86877310-credential-socket\") pod \"27d14b9a-3320-4e79-85b6-662d86877310\" (UID: \"27d14b9a-3320-4e79-85b6-662d86877310\") " Nov 03 01:46:09.575158 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:09.575115 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71058df3-a083-440b-b68a-5dd686509a66-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71058df3-a083-440b-b68a-5dd686509a66-istio-data\") pod \"71058df3-a083-440b-b68a-5dd686509a66\" (UID: \"71058df3-a083-440b-b68a-5dd686509a66\") " Nov 03 01:46:09.575262 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:09.575191 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27d14b9a-3320-4e79-85b6-662d86877310-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27d14b9a-3320-4e79-85b6-662d86877310-istio-data\") pod \"27d14b9a-3320-4e79-85b6-662d86877310\" (UID: \"27d14b9a-3320-4e79-85b6-662d86877310\") " Nov 03 01:46:09.575385 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:09.575344 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27d14b9a-3320-4e79-85b6-662d86877310-istiod-ca-cert" pod "27d14b9a-3320-4e79-85b6-662d86877310" (UID: "27d14b9a-3320-4e79-85b6-662d86877310"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:46:09.576069 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:09.575743 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71058df3-a083-440b-b68a-5dd686509a66-istiod-ca-cert" pod "71058df3-a083-440b-b68a-5dd686509a66" (UID: "71058df3-a083-440b-b68a-5dd686509a66"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:46:09.576069 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:09.575845 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27d14b9a-3320-4e79-85b6-662d86877310-credential-socket" pod "27d14b9a-3320-4e79-85b6-662d86877310" (UID: "27d14b9a-3320-4e79-85b6-662d86877310"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:46:09.576069 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:09.575867 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71058df3-a083-440b-b68a-5dd686509a66-istio-data" pod "71058df3-a083-440b-b68a-5dd686509a66" (UID: "71058df3-a083-440b-b68a-5dd686509a66"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:46:09.576069 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:09.575905 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27d14b9a-3320-4e79-85b6-662d86877310-workload-socket" pod "27d14b9a-3320-4e79-85b6-662d86877310" (UID: "27d14b9a-3320-4e79-85b6-662d86877310"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:46:09.576069 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:09.576042 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27d14b9a-3320-4e79-85b6-662d86877310-istio-data" pod "27d14b9a-3320-4e79-85b6-662d86877310" (UID: "27d14b9a-3320-4e79-85b6-662d86877310"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:46:09.576445 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:09.576198 5437 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71058df3-a083-440b-b68a-5dd686509a66-credential-socket\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:46:09.576445 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:09.576222 5437 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71058df3-a083-440b-b68a-5dd686509a66-workload-socket\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:46:09.576445 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:09.576237 5437 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71058df3-a083-440b-b68a-5dd686509a66-workload-certs\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:46:09.576445 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:09.576251 5437 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27d14b9a-3320-4e79-85b6-662d86877310-workload-certs\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:46:09.576445 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:09.576264 5437 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71058df3-a083-440b-b68a-5dd686509a66-istiod-ca-cert\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:46:09.576445 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:09.576278 5437 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27d14b9a-3320-4e79-85b6-662d86877310-istiod-ca-cert\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:46:09.576445 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:09.576291 5437 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27d14b9a-3320-4e79-85b6-662d86877310-workload-socket\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:46:09.576445 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:09.576305 5437 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27d14b9a-3320-4e79-85b6-662d86877310-credential-socket\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:46:09.576445 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:09.576318 5437 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71058df3-a083-440b-b68a-5dd686509a66-istio-data\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:46:09.577031 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:09.576993 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71058df3-a083-440b-b68a-5dd686509a66-kube-api-access-2h22m" pod "71058df3-a083-440b-b68a-5dd686509a66" (UID: "71058df3-a083-440b-b68a-5dd686509a66"). InnerVolumeSpecName "kube-api-access-2h22m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:46:09.577355 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:09.577331 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27d14b9a-3320-4e79-85b6-662d86877310-istio-token" pod "27d14b9a-3320-4e79-85b6-662d86877310" (UID: "27d14b9a-3320-4e79-85b6-662d86877310"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:46:09.577507 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:09.577429 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71058df3-a083-440b-b68a-5dd686509a66-istio-token" pod "71058df3-a083-440b-b68a-5dd686509a66" (UID: "71058df3-a083-440b-b68a-5dd686509a66"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:46:09.577507 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:09.577477 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/71058df3-a083-440b-b68a-5dd686509a66-istio-podinfo" pod "71058df3-a083-440b-b68a-5dd686509a66" (UID: "71058df3-a083-440b-b68a-5dd686509a66"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:46:09.577755 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:09.577732 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71058df3-a083-440b-b68a-5dd686509a66-istio-envoy" pod "71058df3-a083-440b-b68a-5dd686509a66" (UID: "71058df3-a083-440b-b68a-5dd686509a66"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:46:09.577880 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:09.577865 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27d14b9a-3320-4e79-85b6-662d86877310-istio-envoy" pod "27d14b9a-3320-4e79-85b6-662d86877310" (UID: "27d14b9a-3320-4e79-85b6-662d86877310"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:46:09.578281 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:09.578252 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/27d14b9a-3320-4e79-85b6-662d86877310-istio-podinfo" pod "27d14b9a-3320-4e79-85b6-662d86877310" (UID: "27d14b9a-3320-4e79-85b6-662d86877310"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:46:09.578802 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:09.578783 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27d14b9a-3320-4e79-85b6-662d86877310-kube-api-access-pjz8c" pod "27d14b9a-3320-4e79-85b6-662d86877310" (UID: "27d14b9a-3320-4e79-85b6-662d86877310"). InnerVolumeSpecName "kube-api-access-pjz8c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:46:09.678125 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:09.678086 5437 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27d14b9a-3320-4e79-85b6-662d86877310-istio-data\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:46:09.678125 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:09.678122 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2h22m\" (UniqueName: \"kubernetes.io/projected/71058df3-a083-440b-b68a-5dd686509a66-kube-api-access-2h22m\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:46:09.678125 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:09.678133 5437 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71058df3-a083-440b-b68a-5dd686509a66-istio-podinfo\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:46:09.678331 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:09.678143 5437 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27d14b9a-3320-4e79-85b6-662d86877310-istio-token\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:46:09.678331 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:09.678152 5437 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71058df3-a083-440b-b68a-5dd686509a66-istio-envoy\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:46:09.678331 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:09.678160 5437 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71058df3-a083-440b-b68a-5dd686509a66-istio-token\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:46:09.678331 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:09.678169 5437 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27d14b9a-3320-4e79-85b6-662d86877310-istio-podinfo\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:46:09.678331 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:09.678178 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjz8c\" (UniqueName: \"kubernetes.io/projected/27d14b9a-3320-4e79-85b6-662d86877310-kube-api-access-pjz8c\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:46:09.678331 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:09.678187 5437 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27d14b9a-3320-4e79-85b6-662d86877310-istio-envoy\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:46:10.007534 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:10.007492 5437 generic.go:356] "Generic (PLEG): container finished" podID="27d14b9a-3320-4e79-85b6-662d86877310" containerID="32940d908f0602e5c9546989b63eb6cd41bd35095d5bf1dcea6a5e8d190b5052" exitCode=0 Nov 03 01:46:10.007757 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:10.007561 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-gh4dm" Nov 03 01:46:10.007757 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:10.007572 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-gh4dm" event={"ID":"27d14b9a-3320-4e79-85b6-662d86877310","Type":"ContainerDied","Data":"32940d908f0602e5c9546989b63eb6cd41bd35095d5bf1dcea6a5e8d190b5052"} Nov 03 01:46:10.007757 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:10.007625 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-gh4dm" event={"ID":"27d14b9a-3320-4e79-85b6-662d86877310","Type":"ContainerDied","Data":"c0a1967e50f7061548eb91240a2ec5f3d7dab900c9b62819b2d774a267e6449b"} Nov 03 01:46:10.007757 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:10.007660 5437 scope.go:122] "RemoveContainer" containerID="32940d908f0602e5c9546989b63eb6cd41bd35095d5bf1dcea6a5e8d190b5052" Nov 03 01:46:10.009547 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:10.009518 5437 generic.go:356] "Generic (PLEG): container finished" podID="71058df3-a083-440b-b68a-5dd686509a66" containerID="21cf4f2e51142980b59d8e8f7f18a274a46eeb54c6305d7703ae7e12f38862dd" exitCode=0 Nov 03 01:46:10.009681 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:10.009580 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-m26wn" event={"ID":"71058df3-a083-440b-b68a-5dd686509a66","Type":"ContainerDied","Data":"21cf4f2e51142980b59d8e8f7f18a274a46eeb54c6305d7703ae7e12f38862dd"} Nov 03 01:46:10.009681 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:10.009620 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-m26wn" event={"ID":"71058df3-a083-440b-b68a-5dd686509a66","Type":"ContainerDied","Data":"c0330276fc2a0b5554fb57658cb6006f4b2f3c1d1676c16f2634628d707171bf"} Nov 03 01:46:10.009799 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:10.009731 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-m26wn" Nov 03 01:46:10.018222 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:10.018201 5437 scope.go:122] "RemoveContainer" containerID="32940d908f0602e5c9546989b63eb6cd41bd35095d5bf1dcea6a5e8d190b5052" Nov 03 01:46:10.018641 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:46:10.018612 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32940d908f0602e5c9546989b63eb6cd41bd35095d5bf1dcea6a5e8d190b5052\": container with ID starting with 32940d908f0602e5c9546989b63eb6cd41bd35095d5bf1dcea6a5e8d190b5052 not found: ID does not exist" containerID="32940d908f0602e5c9546989b63eb6cd41bd35095d5bf1dcea6a5e8d190b5052" Nov 03 01:46:10.018748 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:10.018647 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32940d908f0602e5c9546989b63eb6cd41bd35095d5bf1dcea6a5e8d190b5052"} err="failed to get container status \"32940d908f0602e5c9546989b63eb6cd41bd35095d5bf1dcea6a5e8d190b5052\": rpc error: code = NotFound desc = could not find container \"32940d908f0602e5c9546989b63eb6cd41bd35095d5bf1dcea6a5e8d190b5052\": container with ID starting with 32940d908f0602e5c9546989b63eb6cd41bd35095d5bf1dcea6a5e8d190b5052 not found: ID does not exist" Nov 03 01:46:10.018748 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:10.018664 5437 scope.go:122] "RemoveContainer" containerID="21cf4f2e51142980b59d8e8f7f18a274a46eeb54c6305d7703ae7e12f38862dd" Nov 03 01:46:10.029255 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:10.029234 5437 scope.go:122] "RemoveContainer" containerID="21cf4f2e51142980b59d8e8f7f18a274a46eeb54c6305d7703ae7e12f38862dd" Nov 03 01:46:10.029584 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:46:10.029562 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21cf4f2e51142980b59d8e8f7f18a274a46eeb54c6305d7703ae7e12f38862dd\": container with ID starting with 21cf4f2e51142980b59d8e8f7f18a274a46eeb54c6305d7703ae7e12f38862dd not found: ID does not exist" containerID="21cf4f2e51142980b59d8e8f7f18a274a46eeb54c6305d7703ae7e12f38862dd" Nov 03 01:46:10.029680 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:10.029591 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21cf4f2e51142980b59d8e8f7f18a274a46eeb54c6305d7703ae7e12f38862dd"} err="failed to get container status \"21cf4f2e51142980b59d8e8f7f18a274a46eeb54c6305d7703ae7e12f38862dd\": rpc error: code = NotFound desc = could not find container \"21cf4f2e51142980b59d8e8f7f18a274a46eeb54c6305d7703ae7e12f38862dd\": container with ID starting with 21cf4f2e51142980b59d8e8f7f18a274a46eeb54c6305d7703ae7e12f38862dd not found: ID does not exist" Nov 03 01:46:10.181331 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:10.181295 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-m26wn"] Nov 03 01:46:10.188022 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:10.187993 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-m26wn"] Nov 03 01:46:10.301070 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:10.301038 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-gh4dm"] Nov 03 01:46:10.304307 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:10.304277 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-gh4dm"] Nov 03 01:46:10.462386 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:10.458524 5437 patch_prober.go:29] interesting pod/e2e-worker-w2-9da431-openshift-default-65c799fd77-m26wn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 01:46:10.462386 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:10.458598 5437 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-m26wn" podUID="71058df3-a083-440b-b68a-5dd686509a66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 01:46:11.017835 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:11.017785 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:46:12.090282 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:12.090246 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27d14b9a-3320-4e79-85b6-662d86877310" path="/var/lib/kubelet/pods/27d14b9a-3320-4e79-85b6-662d86877310/volumes" Nov 03 01:46:12.090815 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:12.090800 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71058df3-a083-440b-b68a-5dd686509a66" path="/var/lib/kubelet/pods/71058df3-a083-440b-b68a-5dd686509a66/volumes" Nov 03 01:46:14.037833 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:46:14.037796 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 01:46:16.048307 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:16.048249 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:46:16.268005 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:16.267971 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:46:20.517440 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:20.517396 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mdjkm"] Nov 03 01:46:20.517864 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:20.517676 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mdjkm" podUID="be9a9975-2bf4-47b3-918f-6ef9e4a88bca" containerName="istio-proxy" containerID="cri-o://0e156e7450453f8c7398bec9f3b615389046481461ea0840f94f806e34ff864d" gracePeriod=30 Nov 03 01:46:21.079802 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:21.078081 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:46:25.792201 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:25.792121 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mdjkm" Nov 03 01:46:25.821891 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:25.821852 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be9a9975-2bf4-47b3-918f-6ef9e4a88bca-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be9a9975-2bf4-47b3-918f-6ef9e4a88bca-istiod-ca-cert\") pod \"be9a9975-2bf4-47b3-918f-6ef9e4a88bca\" (UID: \"be9a9975-2bf4-47b3-918f-6ef9e4a88bca\") " Nov 03 01:46:25.822100 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:25.821902 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be9a9975-2bf4-47b3-918f-6ef9e4a88bca-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be9a9975-2bf4-47b3-918f-6ef9e4a88bca-workload-socket\") pod \"be9a9975-2bf4-47b3-918f-6ef9e4a88bca\" (UID: \"be9a9975-2bf4-47b3-918f-6ef9e4a88bca\") " Nov 03 01:46:25.822100 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:25.821931 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be9a9975-2bf4-47b3-918f-6ef9e4a88bca-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be9a9975-2bf4-47b3-918f-6ef9e4a88bca-istio-data\") pod \"be9a9975-2bf4-47b3-918f-6ef9e4a88bca\" (UID: \"be9a9975-2bf4-47b3-918f-6ef9e4a88bca\") " Nov 03 01:46:25.822100 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:25.821967 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be9a9975-2bf4-47b3-918f-6ef9e4a88bca-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be9a9975-2bf4-47b3-918f-6ef9e4a88bca-istio-envoy\") pod \"be9a9975-2bf4-47b3-918f-6ef9e4a88bca\" (UID: \"be9a9975-2bf4-47b3-918f-6ef9e4a88bca\") " Nov 03 01:46:25.822100 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:25.822043 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be9a9975-2bf4-47b3-918f-6ef9e4a88bca-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be9a9975-2bf4-47b3-918f-6ef9e4a88bca-workload-certs\") pod \"be9a9975-2bf4-47b3-918f-6ef9e4a88bca\" (UID: \"be9a9975-2bf4-47b3-918f-6ef9e4a88bca\") " Nov 03 01:46:25.822304 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:25.822120 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be9a9975-2bf4-47b3-918f-6ef9e4a88bca-istio-token\" (UniqueName: \"kubernetes.io/projected/be9a9975-2bf4-47b3-918f-6ef9e4a88bca-istio-token\") pod \"be9a9975-2bf4-47b3-918f-6ef9e4a88bca\" (UID: \"be9a9975-2bf4-47b3-918f-6ef9e4a88bca\") " Nov 03 01:46:25.822304 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:25.822207 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/be9a9975-2bf4-47b3-918f-6ef9e4a88bca-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be9a9975-2bf4-47b3-918f-6ef9e4a88bca-istio-podinfo\") pod \"be9a9975-2bf4-47b3-918f-6ef9e4a88bca\" (UID: \"be9a9975-2bf4-47b3-918f-6ef9e4a88bca\") " Nov 03 01:46:25.822304 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:25.822267 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be9a9975-2bf4-47b3-918f-6ef9e4a88bca-kube-api-access-464nr\" (UniqueName: \"kubernetes.io/projected/be9a9975-2bf4-47b3-918f-6ef9e4a88bca-kube-api-access-464nr\") pod \"be9a9975-2bf4-47b3-918f-6ef9e4a88bca\" (UID: \"be9a9975-2bf4-47b3-918f-6ef9e4a88bca\") " Nov 03 01:46:25.822542 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:25.822322 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be9a9975-2bf4-47b3-918f-6ef9e4a88bca-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be9a9975-2bf4-47b3-918f-6ef9e4a88bca-credential-socket\") pod \"be9a9975-2bf4-47b3-918f-6ef9e4a88bca\" (UID: \"be9a9975-2bf4-47b3-918f-6ef9e4a88bca\") " Nov 03 01:46:25.822542 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:25.822262 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be9a9975-2bf4-47b3-918f-6ef9e4a88bca-workload-socket" pod "be9a9975-2bf4-47b3-918f-6ef9e4a88bca" (UID: "be9a9975-2bf4-47b3-918f-6ef9e4a88bca"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:46:25.822542 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:25.822284 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be9a9975-2bf4-47b3-918f-6ef9e4a88bca-istio-data" pod "be9a9975-2bf4-47b3-918f-6ef9e4a88bca" (UID: "be9a9975-2bf4-47b3-918f-6ef9e4a88bca"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:46:25.822542 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:25.822433 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be9a9975-2bf4-47b3-918f-6ef9e4a88bca-workload-certs" pod "be9a9975-2bf4-47b3-918f-6ef9e4a88bca" (UID: "be9a9975-2bf4-47b3-918f-6ef9e4a88bca"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:46:25.822542 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:25.822355 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be9a9975-2bf4-47b3-918f-6ef9e4a88bca-istiod-ca-cert" pod "be9a9975-2bf4-47b3-918f-6ef9e4a88bca" (UID: "be9a9975-2bf4-47b3-918f-6ef9e4a88bca"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:46:25.822796 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:25.822638 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be9a9975-2bf4-47b3-918f-6ef9e4a88bca-credential-socket" pod "be9a9975-2bf4-47b3-918f-6ef9e4a88bca" (UID: "be9a9975-2bf4-47b3-918f-6ef9e4a88bca"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:46:25.823036 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:25.823016 5437 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be9a9975-2bf4-47b3-918f-6ef9e4a88bca-credential-socket\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:46:25.823111 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:25.823040 5437 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be9a9975-2bf4-47b3-918f-6ef9e4a88bca-istiod-ca-cert\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:46:25.823111 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:25.823056 5437 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be9a9975-2bf4-47b3-918f-6ef9e4a88bca-workload-socket\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:46:25.823111 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:25.823070 5437 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be9a9975-2bf4-47b3-918f-6ef9e4a88bca-istio-data\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:46:25.823111 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:25.823084 5437 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be9a9975-2bf4-47b3-918f-6ef9e4a88bca-workload-certs\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:46:25.824411 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:25.824386 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be9a9975-2bf4-47b3-918f-6ef9e4a88bca-kube-api-access-464nr" pod "be9a9975-2bf4-47b3-918f-6ef9e4a88bca" (UID: "be9a9975-2bf4-47b3-918f-6ef9e4a88bca"). InnerVolumeSpecName "kube-api-access-464nr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:46:25.824411 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:25.824402 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be9a9975-2bf4-47b3-918f-6ef9e4a88bca-istio-token" pod "be9a9975-2bf4-47b3-918f-6ef9e4a88bca" (UID: "be9a9975-2bf4-47b3-918f-6ef9e4a88bca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:46:25.824748 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:25.824732 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be9a9975-2bf4-47b3-918f-6ef9e4a88bca-istio-envoy" pod "be9a9975-2bf4-47b3-918f-6ef9e4a88bca" (UID: "be9a9975-2bf4-47b3-918f-6ef9e4a88bca"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:46:25.824813 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:25.824771 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/be9a9975-2bf4-47b3-918f-6ef9e4a88bca-istio-podinfo" pod "be9a9975-2bf4-47b3-918f-6ef9e4a88bca" (UID: "be9a9975-2bf4-47b3-918f-6ef9e4a88bca"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:46:25.896069 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:25.896020 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:46:25.924003 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:25.923966 5437 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be9a9975-2bf4-47b3-918f-6ef9e4a88bca-istio-envoy\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:46:25.924003 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:25.923998 5437 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be9a9975-2bf4-47b3-918f-6ef9e4a88bca-istio-token\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:46:25.924003 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:25.924009 5437 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be9a9975-2bf4-47b3-918f-6ef9e4a88bca-istio-podinfo\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:46:25.924248 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:25.924022 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-464nr\" (UniqueName: \"kubernetes.io/projected/be9a9975-2bf4-47b3-918f-6ef9e4a88bca-kube-api-access-464nr\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:46:26.133155 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:26.133123 5437 generic.go:356] "Generic (PLEG): container finished" podID="be9a9975-2bf4-47b3-918f-6ef9e4a88bca" containerID="0e156e7450453f8c7398bec9f3b615389046481461ea0840f94f806e34ff864d" exitCode=0 Nov 03 01:46:26.133326 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:26.133173 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mdjkm" Nov 03 01:46:26.133326 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:26.133205 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mdjkm" event={"ID":"be9a9975-2bf4-47b3-918f-6ef9e4a88bca","Type":"ContainerDied","Data":"0e156e7450453f8c7398bec9f3b615389046481461ea0840f94f806e34ff864d"} Nov 03 01:46:26.133326 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:26.133268 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mdjkm" event={"ID":"be9a9975-2bf4-47b3-918f-6ef9e4a88bca","Type":"ContainerDied","Data":"e2a5ced8093ae060ec1f953bd67da6cf0ce74bc93dd3f0b48cf3b76bb6395855"} Nov 03 01:46:26.133326 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:26.133295 5437 scope.go:122] "RemoveContainer" containerID="0e156e7450453f8c7398bec9f3b615389046481461ea0840f94f806e34ff864d" Nov 03 01:46:26.143493 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:26.143466 5437 scope.go:122] "RemoveContainer" containerID="0e156e7450453f8c7398bec9f3b615389046481461ea0840f94f806e34ff864d" Nov 03 01:46:26.143877 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:46:26.143844 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e156e7450453f8c7398bec9f3b615389046481461ea0840f94f806e34ff864d\": container with ID starting with 0e156e7450453f8c7398bec9f3b615389046481461ea0840f94f806e34ff864d not found: ID does not exist" containerID="0e156e7450453f8c7398bec9f3b615389046481461ea0840f94f806e34ff864d" Nov 03 01:46:26.143977 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:26.143884 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e156e7450453f8c7398bec9f3b615389046481461ea0840f94f806e34ff864d"} err="failed to get container status \"0e156e7450453f8c7398bec9f3b615389046481461ea0840f94f806e34ff864d\": rpc error: code = NotFound desc = could not find container \"0e156e7450453f8c7398bec9f3b615389046481461ea0840f94f806e34ff864d\": container with ID starting with 0e156e7450453f8c7398bec9f3b615389046481461ea0840f94f806e34ff864d not found: ID does not exist" Nov 03 01:46:26.307830 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:26.307760 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mdjkm"] Nov 03 01:46:26.311726 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:26.311694 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mdjkm"] Nov 03 01:46:28.090729 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:28.090690 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be9a9975-2bf4-47b3-918f-6ef9e4a88bca" path="/var/lib/kubelet/pods/be9a9975-2bf4-47b3-918f-6ef9e4a88bca/volumes" Nov 03 01:46:30.802042 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:30.801995 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:46:34.347484 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:34.347439 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:46:35.535907 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:35.535868 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:46:35.884767 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:35.884731 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:46:40.803397 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:40.797048 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:46:42.244563 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:42.244523 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:46:43.506973 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:43.506932 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:46:43.708504 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:43.708455 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:46:44.036557 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:46:44.036520 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 01:46:45.206249 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:45.206213 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:46:45.994442 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:45.994408 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:46:48.359028 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:48.358986 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:46:50.981158 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:50.981104 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:46:55.776431 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:55.776079 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:46:55.985814 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:55.985776 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:46:57.260179 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:57.260137 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:46:58.885110 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:58.885065 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lrcmd"] Nov 03 01:46:58.885932 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:58.885910 5437 cpu_manager.go:414] "containerMap: removing container" podUID="71058df3-a083-440b-b68a-5dd686509a66" containerName="istio-proxy" Nov 03 01:46:58.886033 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:58.885932 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71058df3-a083-440b-b68a-5dd686509a66" containerName="istio-proxy" Nov 03 01:46:58.886033 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:58.885950 5437 cpu_manager.go:414] "containerMap: removing container" podUID="18705810-dd2e-41bc-886a-f8ec91844c8f" containerName="istio-proxy" Nov 03 01:46:58.886033 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:58.885958 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18705810-dd2e-41bc-886a-f8ec91844c8f" containerName="istio-proxy" Nov 03 01:46:58.886033 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:58.885997 5437 cpu_manager.go:414] "containerMap: removing container" podUID="27d14b9a-3320-4e79-85b6-662d86877310" containerName="istio-proxy" Nov 03 01:46:58.886033 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:58.886005 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27d14b9a-3320-4e79-85b6-662d86877310" containerName="istio-proxy" Nov 03 01:46:58.886283 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:58.886051 5437 cpu_manager.go:414] "containerMap: removing container" podUID="c3f8a3e9-ae5d-4029-8fb1-633693e63ac1" containerName="cleanup" Nov 03 01:46:58.886283 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:58.886059 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3f8a3e9-ae5d-4029-8fb1-633693e63ac1" containerName="cleanup" Nov 03 01:46:58.886283 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:58.886075 5437 cpu_manager.go:414] "containerMap: removing container" podUID="de0ef434-0294-4dba-8112-317298a2acb5" containerName="istio-proxy" Nov 03 01:46:58.886283 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:58.886083 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de0ef434-0294-4dba-8112-317298a2acb5" containerName="istio-proxy" Nov 03 01:46:58.886283 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:58.886100 5437 cpu_manager.go:414] "containerMap: removing container" podUID="be9a9975-2bf4-47b3-918f-6ef9e4a88bca" containerName="istio-proxy" Nov 03 01:46:58.886283 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:58.886107 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be9a9975-2bf4-47b3-918f-6ef9e4a88bca" containerName="istio-proxy" Nov 03 01:46:58.886283 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:58.886253 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="de0ef434-0294-4dba-8112-317298a2acb5" containerName="istio-proxy" Nov 03 01:46:58.886283 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:58.886269 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="be9a9975-2bf4-47b3-918f-6ef9e4a88bca" containerName="istio-proxy" Nov 03 01:46:58.886283 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:58.886282 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="71058df3-a083-440b-b68a-5dd686509a66" containerName="istio-proxy" Nov 03 01:46:58.886727 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:58.886290 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="18705810-dd2e-41bc-886a-f8ec91844c8f" containerName="istio-proxy" Nov 03 01:46:58.886727 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:58.886301 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3f8a3e9-ae5d-4029-8fb1-633693e63ac1" containerName="cleanup" Nov 03 01:46:58.886727 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:58.886314 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="27d14b9a-3320-4e79-85b6-662d86877310" containerName="istio-proxy" Nov 03 01:46:58.889943 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:58.889918 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lrcmd" Nov 03 01:46:58.944074 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:58.944037 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-2b9f22c1-openshift-default-dockercfg-p5xl7\"" Nov 03 01:46:59.049180 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:59.049130 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/296123df-fd9f-4268-a61c-bb6ee52240be-istio-podinfo\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lrcmd\" (UID: \"296123df-fd9f-4268-a61c-bb6ee52240be\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lrcmd" Nov 03 01:46:59.049415 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:59.049252 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/296123df-fd9f-4268-a61c-bb6ee52240be-istio-data\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lrcmd\" (UID: \"296123df-fd9f-4268-a61c-bb6ee52240be\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lrcmd" Nov 03 01:46:59.049415 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:59.049357 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/296123df-fd9f-4268-a61c-bb6ee52240be-credential-socket\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lrcmd\" (UID: \"296123df-fd9f-4268-a61c-bb6ee52240be\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lrcmd" Nov 03 01:46:59.049515 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:59.049487 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/296123df-fd9f-4268-a61c-bb6ee52240be-istio-envoy\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lrcmd\" (UID: \"296123df-fd9f-4268-a61c-bb6ee52240be\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lrcmd" Nov 03 01:46:59.049515 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:59.049505 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/296123df-fd9f-4268-a61c-bb6ee52240be-istio-token\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lrcmd\" (UID: \"296123df-fd9f-4268-a61c-bb6ee52240be\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lrcmd" Nov 03 01:46:59.049598 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:59.049536 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/296123df-fd9f-4268-a61c-bb6ee52240be-workload-certs\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lrcmd\" (UID: \"296123df-fd9f-4268-a61c-bb6ee52240be\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lrcmd" Nov 03 01:46:59.049656 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:59.049637 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgb6h\" (UniqueName: \"kubernetes.io/projected/296123df-fd9f-4268-a61c-bb6ee52240be-kube-api-access-wgb6h\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lrcmd\" (UID: \"296123df-fd9f-4268-a61c-bb6ee52240be\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lrcmd" Nov 03 01:46:59.049760 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:59.049748 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/296123df-fd9f-4268-a61c-bb6ee52240be-workload-socket\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lrcmd\" (UID: \"296123df-fd9f-4268-a61c-bb6ee52240be\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lrcmd" Nov 03 01:46:59.049883 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:59.049768 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/296123df-fd9f-4268-a61c-bb6ee52240be-istiod-ca-cert\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lrcmd\" (UID: \"296123df-fd9f-4268-a61c-bb6ee52240be\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lrcmd" Nov 03 01:46:59.089595 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:59.089561 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lrcmd"] Nov 03 01:46:59.151123 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:59.151031 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/296123df-fd9f-4268-a61c-bb6ee52240be-istio-podinfo\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lrcmd\" (UID: \"296123df-fd9f-4268-a61c-bb6ee52240be\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lrcmd" Nov 03 01:46:59.151123 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:59.151090 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/296123df-fd9f-4268-a61c-bb6ee52240be-istio-data\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lrcmd\" (UID: \"296123df-fd9f-4268-a61c-bb6ee52240be\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lrcmd" Nov 03 01:46:59.151402 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:59.151141 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/296123df-fd9f-4268-a61c-bb6ee52240be-credential-socket\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lrcmd\" (UID: \"296123df-fd9f-4268-a61c-bb6ee52240be\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lrcmd" Nov 03 01:46:59.151402 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:59.151202 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/296123df-fd9f-4268-a61c-bb6ee52240be-istio-envoy\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lrcmd\" (UID: \"296123df-fd9f-4268-a61c-bb6ee52240be\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lrcmd" Nov 03 01:46:59.151402 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:59.151226 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/296123df-fd9f-4268-a61c-bb6ee52240be-istio-token\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lrcmd\" (UID: \"296123df-fd9f-4268-a61c-bb6ee52240be\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lrcmd" Nov 03 01:46:59.151402 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:59.151268 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/296123df-fd9f-4268-a61c-bb6ee52240be-workload-certs\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lrcmd\" (UID: \"296123df-fd9f-4268-a61c-bb6ee52240be\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lrcmd" Nov 03 01:46:59.151402 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:59.151330 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgb6h\" (UniqueName: \"kubernetes.io/projected/296123df-fd9f-4268-a61c-bb6ee52240be-kube-api-access-wgb6h\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lrcmd\" (UID: \"296123df-fd9f-4268-a61c-bb6ee52240be\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lrcmd" Nov 03 01:46:59.151679 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:59.151490 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/296123df-fd9f-4268-a61c-bb6ee52240be-workload-socket\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lrcmd\" (UID: \"296123df-fd9f-4268-a61c-bb6ee52240be\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lrcmd" Nov 03 01:46:59.151679 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:59.151518 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/296123df-fd9f-4268-a61c-bb6ee52240be-istiod-ca-cert\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lrcmd\" (UID: \"296123df-fd9f-4268-a61c-bb6ee52240be\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lrcmd" Nov 03 01:46:59.151679 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:59.151632 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/296123df-fd9f-4268-a61c-bb6ee52240be-istio-data\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lrcmd\" (UID: \"296123df-fd9f-4268-a61c-bb6ee52240be\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lrcmd" Nov 03 01:46:59.151854 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:59.151707 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/296123df-fd9f-4268-a61c-bb6ee52240be-credential-socket\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lrcmd\" (UID: \"296123df-fd9f-4268-a61c-bb6ee52240be\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lrcmd" Nov 03 01:46:59.151986 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:59.151960 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/296123df-fd9f-4268-a61c-bb6ee52240be-workload-certs\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lrcmd\" (UID: \"296123df-fd9f-4268-a61c-bb6ee52240be\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lrcmd" Nov 03 01:46:59.152140 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:59.152116 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/296123df-fd9f-4268-a61c-bb6ee52240be-istiod-ca-cert\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lrcmd\" (UID: \"296123df-fd9f-4268-a61c-bb6ee52240be\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lrcmd" Nov 03 01:46:59.152251 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:59.152213 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/296123df-fd9f-4268-a61c-bb6ee52240be-workload-socket\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lrcmd\" (UID: \"296123df-fd9f-4268-a61c-bb6ee52240be\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lrcmd" Nov 03 01:46:59.153595 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:59.153575 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/296123df-fd9f-4268-a61c-bb6ee52240be-istio-envoy\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lrcmd\" (UID: \"296123df-fd9f-4268-a61c-bb6ee52240be\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lrcmd" Nov 03 01:46:59.153717 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:59.153702 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/296123df-fd9f-4268-a61c-bb6ee52240be-istio-podinfo\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lrcmd\" (UID: \"296123df-fd9f-4268-a61c-bb6ee52240be\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lrcmd" Nov 03 01:46:59.214172 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:59.214134 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/296123df-fd9f-4268-a61c-bb6ee52240be-istio-token\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lrcmd\" (UID: \"296123df-fd9f-4268-a61c-bb6ee52240be\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lrcmd" Nov 03 01:46:59.214456 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:59.214429 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgb6h\" (UniqueName: \"kubernetes.io/projected/296123df-fd9f-4268-a61c-bb6ee52240be-kube-api-access-wgb6h\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lrcmd\" (UID: \"296123df-fd9f-4268-a61c-bb6ee52240be\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lrcmd" Nov 03 01:46:59.505194 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:59.505083 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lrcmd" Nov 03 01:46:59.573987 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:59.573945 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:46:59.971216 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:46:59.971177 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lrcmd"] Nov 03 01:47:00.000011 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:46:59.999978 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod296123df_fd9f_4268_a61c_bb6ee52240be.slice/crio-4eefaed904e675fcfe3645219daa6577a31560fdf12ac5d32af7a7d896caf226 WatchSource:0}: Error finding container 4eefaed904e675fcfe3645219daa6577a31560fdf12ac5d32af7a7d896caf226: Status 404 returned error can't find the container with id 4eefaed904e675fcfe3645219daa6577a31560fdf12ac5d32af7a7d896caf226 Nov 03 01:47:00.002708 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:00.002674 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 01:47:00.002793 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:00.002744 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 01:47:00.002793 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:00.002779 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 01:47:00.398263 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:00.398222 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lrcmd" event={"ID":"296123df-fd9f-4268-a61c-bb6ee52240be","Type":"ContainerStarted","Data":"abb436119f52563e452c26ff2e2478d0a16d35cb488a27a84ed15e12f6ddb21c"} Nov 03 01:47:00.398477 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:00.398289 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lrcmd" event={"ID":"296123df-fd9f-4268-a61c-bb6ee52240be","Type":"ContainerStarted","Data":"4eefaed904e675fcfe3645219daa6577a31560fdf12ac5d32af7a7d896caf226"} Nov 03 01:47:00.506052 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:00.506021 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lrcmd" Nov 03 01:47:00.575851 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:00.575792 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lrcmd" podStartSLOduration=2.575775472 podStartE2EDuration="2.575775472s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:46:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:47:00.573409728 +0000 UTC m=+1702.602892025" watchObservedRunningTime="2026-11-03 01:47:00.575775472 +0000 UTC m=+1702.605257844" Nov 03 01:47:01.082242 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:01.082208 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:47:01.506851 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:01.506762 5437 patch_prober.go:29] interesting pod/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lrcmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 03 01:47:01.506851 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:01.506828 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lrcmd" podUID="296123df-fd9f-4268-a61c-bb6ee52240be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 03 01:47:02.505596 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:02.505553 5437 patch_prober.go:29] interesting pod/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lrcmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 03 01:47:02.506120 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:02.505612 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lrcmd" podUID="296123df-fd9f-4268-a61c-bb6ee52240be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 03 01:47:03.506142 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:03.506104 5437 patch_prober.go:29] interesting pod/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lrcmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 03 01:47:03.506702 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:03.506168 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lrcmd" podUID="296123df-fd9f-4268-a61c-bb6ee52240be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 03 01:47:04.506036 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:04.505993 5437 patch_prober.go:29] interesting pod/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lrcmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 03 01:47:04.506264 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:04.506061 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lrcmd" podUID="296123df-fd9f-4268-a61c-bb6ee52240be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 03 01:47:05.513828 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:05.513791 5437 patch_prober.go:29] interesting pod/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lrcmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 03 01:47:05.514420 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:05.514389 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lrcmd" podUID="296123df-fd9f-4268-a61c-bb6ee52240be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 03 01:47:06.113414 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:06.113354 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:47:06.315070 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:06.315026 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:47:06.505818 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:06.505707 5437 patch_prober.go:29] interesting pod/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lrcmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 03 01:47:06.505818 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:06.505783 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lrcmd" podUID="296123df-fd9f-4268-a61c-bb6ee52240be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 03 01:47:06.552387 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:06.539078 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:47:06.746035 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:06.745993 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:47:07.542230 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:07.542194 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lrcmd" Nov 03 01:47:07.543938 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:07.543913 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lrcmd" Nov 03 01:47:07.547169 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:07.547140 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lrcmd" Nov 03 01:47:10.446958 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:10.446919 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-948470-78b87884df-7hp5g"] Nov 03 01:47:10.447428 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:10.447141 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-948470-78b87884df-7hp5g" podUID="3361c0e5-85a7-466e-8c60-024f958e7227" containerName="maas-api" containerID="cri-o://5b6ec2f29033672dbd525b113db8e28fda3caac8b82e296d087199d828c666b7" gracePeriod=30 Nov 03 01:47:10.570732 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:10.570698 5437 generic.go:356] "Generic (PLEG): container finished" podID="3361c0e5-85a7-466e-8c60-024f958e7227" containerID="5b6ec2f29033672dbd525b113db8e28fda3caac8b82e296d087199d828c666b7" exitCode=0 Nov 03 01:47:10.570904 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:10.570766 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-948470-78b87884df-7hp5g" event={"ID":"3361c0e5-85a7-466e-8c60-024f958e7227","Type":"ContainerDied","Data":"5b6ec2f29033672dbd525b113db8e28fda3caac8b82e296d087199d828c666b7"} Nov 03 01:47:10.602464 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:10.602433 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:47:10.768742 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:10.768713 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-948470-78b87884df-7hp5g" Nov 03 01:47:10.883587 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:10.883552 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3361c0e5-85a7-466e-8c60-024f958e7227-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3361c0e5-85a7-466e-8c60-024f958e7227-metrics-certs\") pod \"3361c0e5-85a7-466e-8c60-024f958e7227\" (UID: \"3361c0e5-85a7-466e-8c60-024f958e7227\") " Nov 03 01:47:10.883779 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:10.883666 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3361c0e5-85a7-466e-8c60-024f958e7227-kube-api-access-z8966\" (UniqueName: \"kubernetes.io/projected/3361c0e5-85a7-466e-8c60-024f958e7227-kube-api-access-z8966\") pod \"3361c0e5-85a7-466e-8c60-024f958e7227\" (UID: \"3361c0e5-85a7-466e-8c60-024f958e7227\") " Nov 03 01:47:10.883911 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:10.883834 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3361c0e5-85a7-466e-8c60-024f958e7227-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3361c0e5-85a7-466e-8c60-024f958e7227-maas-api-tls\") pod \"3361c0e5-85a7-466e-8c60-024f958e7227\" (UID: \"3361c0e5-85a7-466e-8c60-024f958e7227\") " Nov 03 01:47:10.883980 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:10.883940 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3361c0e5-85a7-466e-8c60-024f958e7227-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3361c0e5-85a7-466e-8c60-024f958e7227-openshift-service-ca\") pod \"3361c0e5-85a7-466e-8c60-024f958e7227\" (UID: \"3361c0e5-85a7-466e-8c60-024f958e7227\") " Nov 03 01:47:10.884988 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:10.884295 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3361c0e5-85a7-466e-8c60-024f958e7227-openshift-service-ca" pod "3361c0e5-85a7-466e-8c60-024f958e7227" (UID: "3361c0e5-85a7-466e-8c60-024f958e7227"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:47:10.884988 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:10.884753 5437 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3361c0e5-85a7-466e-8c60-024f958e7227-openshift-service-ca\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:47:10.886468 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:10.886439 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3361c0e5-85a7-466e-8c60-024f958e7227-maas-api-tls" pod "3361c0e5-85a7-466e-8c60-024f958e7227" (UID: "3361c0e5-85a7-466e-8c60-024f958e7227"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:47:10.886603 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:10.886541 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3361c0e5-85a7-466e-8c60-024f958e7227-kube-api-access-z8966" pod "3361c0e5-85a7-466e-8c60-024f958e7227" (UID: "3361c0e5-85a7-466e-8c60-024f958e7227"). InnerVolumeSpecName "kube-api-access-z8966". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:47:10.890534 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:10.890504 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3361c0e5-85a7-466e-8c60-024f958e7227-metrics-certs" pod "3361c0e5-85a7-466e-8c60-024f958e7227" (UID: "3361c0e5-85a7-466e-8c60-024f958e7227"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:47:10.986456 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:10.986414 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8966\" (UniqueName: \"kubernetes.io/projected/3361c0e5-85a7-466e-8c60-024f958e7227-kube-api-access-z8966\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:47:10.986456 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:10.986454 5437 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3361c0e5-85a7-466e-8c60-024f958e7227-maas-api-tls\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:47:10.986668 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:10.986468 5437 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3361c0e5-85a7-466e-8c60-024f958e7227-metrics-certs\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:47:11.578796 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:11.578769 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-948470-78b87884df-7hp5g" Nov 03 01:47:11.579270 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:11.578760 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-948470-78b87884df-7hp5g" event={"ID":"3361c0e5-85a7-466e-8c60-024f958e7227","Type":"ContainerDied","Data":"7fb28eda35b6df34f3a5d3d3b8a43d5a85e4a6a9dddbbb1454d75b8133c4b2a2"} Nov 03 01:47:11.579270 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:11.578902 5437 scope.go:122] "RemoveContainer" containerID="5b6ec2f29033672dbd525b113db8e28fda3caac8b82e296d087199d828c666b7" Nov 03 01:47:11.751775 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:11.751740 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-948470-78b87884df-7hp5g"] Nov 03 01:47:11.802850 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:11.802813 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-948470-78b87884df-7hp5g"] Nov 03 01:47:11.851032 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:11.850947 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:47:12.090434 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:12.090392 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3361c0e5-85a7-466e-8c60-024f958e7227" path="/var/lib/kubelet/pods/3361c0e5-85a7-466e-8c60-024f958e7227/volumes" Nov 03 01:47:12.293987 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:12.293902 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:47:12.504482 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:12.504434 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:47:13.819572 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:13.819527 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:47:13.989224 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:13.989181 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lhxlr"] Nov 03 01:47:13.989931 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:13.989913 5437 cpu_manager.go:414] "containerMap: removing container" podUID="3361c0e5-85a7-466e-8c60-024f958e7227" containerName="maas-api" Nov 03 01:47:13.989981 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:13.989932 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3361c0e5-85a7-466e-8c60-024f958e7227" containerName="maas-api" Nov 03 01:47:13.990038 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:13.990029 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="3361c0e5-85a7-466e-8c60-024f958e7227" containerName="maas-api" Nov 03 01:47:13.995107 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:13.995079 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lhxlr" Nov 03 01:47:14.037667 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:47:14.037626 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 01:47:14.040770 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:14.040744 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-8p8gc"] Nov 03 01:47:14.045101 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:14.045077 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-8p8gc" Nov 03 01:47:14.094645 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:14.094554 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lhxlr"] Nov 03 01:47:14.128373 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:14.128319 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad40b959-18ef-44f8-a81e-d33e19794abf-istio-token\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lhxlr\" (UID: \"ad40b959-18ef-44f8-a81e-d33e19794abf\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lhxlr" Nov 03 01:47:14.128585 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:14.128385 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbpvb\" (UniqueName: \"kubernetes.io/projected/ad40b959-18ef-44f8-a81e-d33e19794abf-kube-api-access-qbpvb\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lhxlr\" (UID: \"ad40b959-18ef-44f8-a81e-d33e19794abf\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lhxlr" Nov 03 01:47:14.128585 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:14.128515 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad40b959-18ef-44f8-a81e-d33e19794abf-credential-socket\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lhxlr\" (UID: \"ad40b959-18ef-44f8-a81e-d33e19794abf\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lhxlr" Nov 03 01:47:14.128715 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:14.128656 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a5060a0-9abf-4586-aef4-8d681a13b2aa-istio-podinfo\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-8p8gc\" (UID: \"3a5060a0-9abf-4586-aef4-8d681a13b2aa\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-8p8gc" Nov 03 01:47:14.128715 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:14.128701 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad40b959-18ef-44f8-a81e-d33e19794abf-istio-podinfo\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lhxlr\" (UID: \"ad40b959-18ef-44f8-a81e-d33e19794abf\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lhxlr" Nov 03 01:47:14.128831 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:14.128811 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad40b959-18ef-44f8-a81e-d33e19794abf-istio-data\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lhxlr\" (UID: \"ad40b959-18ef-44f8-a81e-d33e19794abf\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lhxlr" Nov 03 01:47:14.128893 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:14.128878 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad40b959-18ef-44f8-a81e-d33e19794abf-istiod-ca-cert\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lhxlr\" (UID: \"ad40b959-18ef-44f8-a81e-d33e19794abf\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lhxlr" Nov 03 01:47:14.128934 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:14.128901 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a5060a0-9abf-4586-aef4-8d681a13b2aa-workload-socket\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-8p8gc\" (UID: \"3a5060a0-9abf-4586-aef4-8d681a13b2aa\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-8p8gc" Nov 03 01:47:14.128934 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:14.128925 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfcqq\" (UniqueName: \"kubernetes.io/projected/3a5060a0-9abf-4586-aef4-8d681a13b2aa-kube-api-access-wfcqq\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-8p8gc\" (UID: \"3a5060a0-9abf-4586-aef4-8d681a13b2aa\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-8p8gc" Nov 03 01:47:14.129044 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:14.128987 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a5060a0-9abf-4586-aef4-8d681a13b2aa-istio-envoy\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-8p8gc\" (UID: \"3a5060a0-9abf-4586-aef4-8d681a13b2aa\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-8p8gc" Nov 03 01:47:14.129161 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:14.129143 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a5060a0-9abf-4586-aef4-8d681a13b2aa-credential-socket\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-8p8gc\" (UID: \"3a5060a0-9abf-4586-aef4-8d681a13b2aa\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-8p8gc" Nov 03 01:47:14.129222 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:14.129194 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad40b959-18ef-44f8-a81e-d33e19794abf-workload-certs\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lhxlr\" (UID: \"ad40b959-18ef-44f8-a81e-d33e19794abf\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lhxlr" Nov 03 01:47:14.129222 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:14.129215 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a5060a0-9abf-4586-aef4-8d681a13b2aa-istio-data\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-8p8gc\" (UID: \"3a5060a0-9abf-4586-aef4-8d681a13b2aa\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-8p8gc" Nov 03 01:47:14.129330 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:14.129237 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a5060a0-9abf-4586-aef4-8d681a13b2aa-istiod-ca-cert\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-8p8gc\" (UID: \"3a5060a0-9abf-4586-aef4-8d681a13b2aa\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-8p8gc" Nov 03 01:47:14.129424 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:14.129399 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad40b959-18ef-44f8-a81e-d33e19794abf-workload-socket\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lhxlr\" (UID: \"ad40b959-18ef-44f8-a81e-d33e19794abf\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lhxlr" Nov 03 01:47:14.129489 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:14.129472 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a5060a0-9abf-4586-aef4-8d681a13b2aa-istio-token\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-8p8gc\" (UID: \"3a5060a0-9abf-4586-aef4-8d681a13b2aa\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-8p8gc" Nov 03 01:47:14.129528 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:14.129504 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad40b959-18ef-44f8-a81e-d33e19794abf-istio-envoy\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lhxlr\" (UID: \"ad40b959-18ef-44f8-a81e-d33e19794abf\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lhxlr" Nov 03 01:47:14.129564 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:14.129538 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a5060a0-9abf-4586-aef4-8d681a13b2aa-workload-certs\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-8p8gc\" (UID: \"3a5060a0-9abf-4586-aef4-8d681a13b2aa\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-8p8gc" Nov 03 01:47:14.140687 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:14.140650 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-8p8gc"] Nov 03 01:47:14.231610 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:14.231560 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a5060a0-9abf-4586-aef4-8d681a13b2aa-istio-envoy\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-8p8gc\" (UID: \"3a5060a0-9abf-4586-aef4-8d681a13b2aa\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-8p8gc" Nov 03 01:47:14.231827 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:14.231652 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a5060a0-9abf-4586-aef4-8d681a13b2aa-credential-socket\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-8p8gc\" (UID: \"3a5060a0-9abf-4586-aef4-8d681a13b2aa\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-8p8gc" Nov 03 01:47:14.231827 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:14.231707 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad40b959-18ef-44f8-a81e-d33e19794abf-workload-certs\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lhxlr\" (UID: \"ad40b959-18ef-44f8-a81e-d33e19794abf\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lhxlr" Nov 03 01:47:14.231827 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:14.231734 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a5060a0-9abf-4586-aef4-8d681a13b2aa-istio-data\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-8p8gc\" (UID: \"3a5060a0-9abf-4586-aef4-8d681a13b2aa\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-8p8gc" Nov 03 01:47:14.231827 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:14.231768 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a5060a0-9abf-4586-aef4-8d681a13b2aa-istiod-ca-cert\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-8p8gc\" (UID: \"3a5060a0-9abf-4586-aef4-8d681a13b2aa\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-8p8gc" Nov 03 01:47:14.232053 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:14.231896 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad40b959-18ef-44f8-a81e-d33e19794abf-workload-socket\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lhxlr\" (UID: \"ad40b959-18ef-44f8-a81e-d33e19794abf\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lhxlr" Nov 03 01:47:14.232053 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:14.231998 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a5060a0-9abf-4586-aef4-8d681a13b2aa-istio-token\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-8p8gc\" (UID: \"3a5060a0-9abf-4586-aef4-8d681a13b2aa\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-8p8gc" Nov 03 01:47:14.232053 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:14.232039 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad40b959-18ef-44f8-a81e-d33e19794abf-istio-envoy\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lhxlr\" (UID: \"ad40b959-18ef-44f8-a81e-d33e19794abf\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lhxlr" Nov 03 01:47:14.232207 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:14.232081 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a5060a0-9abf-4586-aef4-8d681a13b2aa-workload-certs\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-8p8gc\" (UID: \"3a5060a0-9abf-4586-aef4-8d681a13b2aa\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-8p8gc" Nov 03 01:47:14.232207 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:14.232116 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad40b959-18ef-44f8-a81e-d33e19794abf-istio-token\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lhxlr\" (UID: \"ad40b959-18ef-44f8-a81e-d33e19794abf\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lhxlr" Nov 03 01:47:14.232207 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:14.232144 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbpvb\" (UniqueName: \"kubernetes.io/projected/ad40b959-18ef-44f8-a81e-d33e19794abf-kube-api-access-qbpvb\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lhxlr\" (UID: \"ad40b959-18ef-44f8-a81e-d33e19794abf\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lhxlr" Nov 03 01:47:14.232207 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:14.232165 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad40b959-18ef-44f8-a81e-d33e19794abf-workload-certs\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lhxlr\" (UID: \"ad40b959-18ef-44f8-a81e-d33e19794abf\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lhxlr" Nov 03 01:47:14.232207 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:14.232167 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a5060a0-9abf-4586-aef4-8d681a13b2aa-istio-data\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-8p8gc\" (UID: \"3a5060a0-9abf-4586-aef4-8d681a13b2aa\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-8p8gc" Nov 03 01:47:14.232207 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:14.232192 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad40b959-18ef-44f8-a81e-d33e19794abf-credential-socket\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lhxlr\" (UID: \"ad40b959-18ef-44f8-a81e-d33e19794abf\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lhxlr" Nov 03 01:47:14.232531 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:14.232325 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a5060a0-9abf-4586-aef4-8d681a13b2aa-istio-podinfo\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-8p8gc\" (UID: \"3a5060a0-9abf-4586-aef4-8d681a13b2aa\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-8p8gc" Nov 03 01:47:14.232531 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:14.232382 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad40b959-18ef-44f8-a81e-d33e19794abf-istio-podinfo\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lhxlr\" (UID: \"ad40b959-18ef-44f8-a81e-d33e19794abf\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lhxlr" Nov 03 01:47:14.232531 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:14.232414 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a5060a0-9abf-4586-aef4-8d681a13b2aa-workload-certs\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-8p8gc\" (UID: \"3a5060a0-9abf-4586-aef4-8d681a13b2aa\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-8p8gc" Nov 03 01:47:14.232531 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:14.232481 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad40b959-18ef-44f8-a81e-d33e19794abf-istio-data\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lhxlr\" (UID: \"ad40b959-18ef-44f8-a81e-d33e19794abf\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lhxlr" Nov 03 01:47:14.232747 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:14.232578 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad40b959-18ef-44f8-a81e-d33e19794abf-istiod-ca-cert\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lhxlr\" (UID: \"ad40b959-18ef-44f8-a81e-d33e19794abf\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lhxlr" Nov 03 01:47:14.232747 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:14.232610 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a5060a0-9abf-4586-aef4-8d681a13b2aa-workload-socket\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-8p8gc\" (UID: \"3a5060a0-9abf-4586-aef4-8d681a13b2aa\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-8p8gc" Nov 03 01:47:14.232747 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:14.232648 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfcqq\" (UniqueName: \"kubernetes.io/projected/3a5060a0-9abf-4586-aef4-8d681a13b2aa-kube-api-access-wfcqq\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-8p8gc\" (UID: \"3a5060a0-9abf-4586-aef4-8d681a13b2aa\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-8p8gc" Nov 03 01:47:14.232900 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:14.232813 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a5060a0-9abf-4586-aef4-8d681a13b2aa-istiod-ca-cert\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-8p8gc\" (UID: \"3a5060a0-9abf-4586-aef4-8d681a13b2aa\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-8p8gc" Nov 03 01:47:14.233126 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:14.233102 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad40b959-18ef-44f8-a81e-d33e19794abf-workload-socket\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lhxlr\" (UID: \"ad40b959-18ef-44f8-a81e-d33e19794abf\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lhxlr" Nov 03 01:47:14.233205 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:14.233122 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a5060a0-9abf-4586-aef4-8d681a13b2aa-credential-socket\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-8p8gc\" (UID: \"3a5060a0-9abf-4586-aef4-8d681a13b2aa\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-8p8gc" Nov 03 01:47:14.233966 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:14.233429 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a5060a0-9abf-4586-aef4-8d681a13b2aa-workload-socket\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-8p8gc\" (UID: \"3a5060a0-9abf-4586-aef4-8d681a13b2aa\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-8p8gc" Nov 03 01:47:14.233966 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:14.233448 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad40b959-18ef-44f8-a81e-d33e19794abf-credential-socket\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lhxlr\" (UID: \"ad40b959-18ef-44f8-a81e-d33e19794abf\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lhxlr" Nov 03 01:47:14.233966 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:14.233546 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad40b959-18ef-44f8-a81e-d33e19794abf-istiod-ca-cert\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lhxlr\" (UID: \"ad40b959-18ef-44f8-a81e-d33e19794abf\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lhxlr" Nov 03 01:47:14.234191 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:14.234165 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad40b959-18ef-44f8-a81e-d33e19794abf-istio-data\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lhxlr\" (UID: \"ad40b959-18ef-44f8-a81e-d33e19794abf\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lhxlr" Nov 03 01:47:14.235805 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:14.235735 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a5060a0-9abf-4586-aef4-8d681a13b2aa-istio-envoy\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-8p8gc\" (UID: \"3a5060a0-9abf-4586-aef4-8d681a13b2aa\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-8p8gc" Nov 03 01:47:14.236500 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:14.236465 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a5060a0-9abf-4586-aef4-8d681a13b2aa-istio-podinfo\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-8p8gc\" (UID: \"3a5060a0-9abf-4586-aef4-8d681a13b2aa\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-8p8gc" Nov 03 01:47:14.236663 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:14.236637 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad40b959-18ef-44f8-a81e-d33e19794abf-istio-envoy\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lhxlr\" (UID: \"ad40b959-18ef-44f8-a81e-d33e19794abf\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lhxlr" Nov 03 01:47:14.236832 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:14.236814 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad40b959-18ef-44f8-a81e-d33e19794abf-istio-podinfo\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lhxlr\" (UID: \"ad40b959-18ef-44f8-a81e-d33e19794abf\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lhxlr" Nov 03 01:47:14.295502 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:14.295469 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfcqq\" (UniqueName: \"kubernetes.io/projected/3a5060a0-9abf-4586-aef4-8d681a13b2aa-kube-api-access-wfcqq\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-8p8gc\" (UID: \"3a5060a0-9abf-4586-aef4-8d681a13b2aa\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-8p8gc" Nov 03 01:47:14.295502 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:14.295494 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a5060a0-9abf-4586-aef4-8d681a13b2aa-istio-token\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-8p8gc\" (UID: \"3a5060a0-9abf-4586-aef4-8d681a13b2aa\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-8p8gc" Nov 03 01:47:14.295722 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:14.295519 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad40b959-18ef-44f8-a81e-d33e19794abf-istio-token\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lhxlr\" (UID: \"ad40b959-18ef-44f8-a81e-d33e19794abf\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lhxlr" Nov 03 01:47:14.295767 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:14.295718 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbpvb\" (UniqueName: \"kubernetes.io/projected/ad40b959-18ef-44f8-a81e-d33e19794abf-kube-api-access-qbpvb\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lhxlr\" (UID: \"ad40b959-18ef-44f8-a81e-d33e19794abf\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lhxlr" Nov 03 01:47:14.316101 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:14.311584 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lhxlr" Nov 03 01:47:14.335152 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:14.335049 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:47:14.362413 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:14.361664 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-8p8gc" Nov 03 01:47:14.913531 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:14.913494 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lhxlr"] Nov 03 01:47:14.916086 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:14.916061 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-8p8gc"] Nov 03 01:47:14.941828 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:47:14.941797 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad40b959_18ef_44f8_a81e_d33e19794abf.slice/crio-50773111940d6c5bb867e9146357f7cb8f777f5531e98bf13b4191325ed9b7f1 WatchSource:0}: Error finding container 50773111940d6c5bb867e9146357f7cb8f777f5531e98bf13b4191325ed9b7f1: Status 404 returned error can't find the container with id 50773111940d6c5bb867e9146357f7cb8f777f5531e98bf13b4191325ed9b7f1 Nov 03 01:47:14.942553 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:47:14.942531 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a5060a0_9abf_4586_aef4_8d681a13b2aa.slice/crio-d45ab1d114c2efd510f9c90364d2be7b2744d017b69a96b40190ce408908528c WatchSource:0}: Error finding container d45ab1d114c2efd510f9c90364d2be7b2744d017b69a96b40190ce408908528c: Status 404 returned error can't find the container with id d45ab1d114c2efd510f9c90364d2be7b2744d017b69a96b40190ce408908528c Nov 03 01:47:14.944448 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:14.944417 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 01:47:14.944521 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:14.944486 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 01:47:14.944521 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:14.944513 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 01:47:14.944878 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:14.944845 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 01:47:14.944946 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:14.944918 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 01:47:14.945067 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:14.944950 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 01:47:15.096451 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:15.096413 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:47:15.608621 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:15.608579 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lhxlr" event={"ID":"ad40b959-18ef-44f8-a81e-d33e19794abf","Type":"ContainerStarted","Data":"56f673e8057177a93349f206f1d2e80bf649739f9f562e6e72f29bda0ad5ce85"} Nov 03 01:47:15.608842 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:15.608677 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lhxlr" event={"ID":"ad40b959-18ef-44f8-a81e-d33e19794abf","Type":"ContainerStarted","Data":"50773111940d6c5bb867e9146357f7cb8f777f5531e98bf13b4191325ed9b7f1"} Nov 03 01:47:15.611079 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:15.611041 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-8p8gc" event={"ID":"3a5060a0-9abf-4586-aef4-8d681a13b2aa","Type":"ContainerStarted","Data":"48cf9404fc106d59f6c5cf3c6cac07d0da8ab76c409859f9b5f7f6319666d8dd"} Nov 03 01:47:15.611232 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:15.611094 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-8p8gc" event={"ID":"3a5060a0-9abf-4586-aef4-8d681a13b2aa","Type":"ContainerStarted","Data":"d45ab1d114c2efd510f9c90364d2be7b2744d017b69a96b40190ce408908528c"} Nov 03 01:47:15.871665 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:15.871540 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lhxlr" podStartSLOduration=2.871525054 podStartE2EDuration="2.871525054s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:47:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:47:15.744745838 +0000 UTC m=+1717.774228136" watchObservedRunningTime="2026-11-03 01:47:15.871525054 +0000 UTC m=+1717.901007357" Nov 03 01:47:15.872849 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:15.872810 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-8p8gc" podStartSLOduration=2.872798272 podStartE2EDuration="2.872798272s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:47:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:47:15.870674166 +0000 UTC m=+1717.900156463" watchObservedRunningTime="2026-11-03 01:47:15.872798272 +0000 UTC m=+1717.902280567" Nov 03 01:47:16.312439 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:16.312400 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lhxlr" Nov 03 01:47:16.314619 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:16.314505 5437 patch_prober.go:29] interesting pod/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lhxlr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Nov 03 01:47:16.314619 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:16.314579 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lhxlr" podUID="ad40b959-18ef-44f8-a81e-d33e19794abf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Nov 03 01:47:16.364584 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:16.364501 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-8p8gc" Nov 03 01:47:16.365310 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:16.365169 5437 patch_prober.go:29] interesting pod/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-8p8gc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Nov 03 01:47:16.365310 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:16.365246 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-8p8gc" podUID="3a5060a0-9abf-4586-aef4-8d681a13b2aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Nov 03 01:47:17.312781 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:17.312701 5437 patch_prober.go:29] interesting pod/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lhxlr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Nov 03 01:47:17.312781 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:17.312755 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lhxlr" podUID="ad40b959-18ef-44f8-a81e-d33e19794abf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Nov 03 01:47:17.366389 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:17.365806 5437 patch_prober.go:29] interesting pod/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-8p8gc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Nov 03 01:47:17.366389 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:17.365885 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-8p8gc" podUID="3a5060a0-9abf-4586-aef4-8d681a13b2aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Nov 03 01:47:17.627455 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:17.617565 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:47:18.312060 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:18.312017 5437 patch_prober.go:29] interesting pod/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lhxlr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Nov 03 01:47:18.312277 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:18.312080 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lhxlr" podUID="ad40b959-18ef-44f8-a81e-d33e19794abf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Nov 03 01:47:18.367602 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:18.367500 5437 patch_prober.go:29] interesting pod/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-8p8gc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Nov 03 01:47:18.367602 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:18.367584 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-8p8gc" podUID="3a5060a0-9abf-4586-aef4-8d681a13b2aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Nov 03 01:47:19.312699 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:19.312659 5437 patch_prober.go:29] interesting pod/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lhxlr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Nov 03 01:47:19.312887 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:19.312720 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lhxlr" podUID="ad40b959-18ef-44f8-a81e-d33e19794abf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Nov 03 01:47:19.365390 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:19.364464 5437 patch_prober.go:29] interesting pod/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-8p8gc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Nov 03 01:47:19.365390 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:19.364526 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-8p8gc" podUID="3a5060a0-9abf-4586-aef4-8d681a13b2aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Nov 03 01:47:19.777022 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:19.776924 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:47:20.272744 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:20.272694 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4952b1-ccfb5b59f-4r2rj"] Nov 03 01:47:20.313297 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:20.313247 5437 patch_prober.go:29] interesting pod/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lhxlr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Nov 03 01:47:20.313603 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:20.313567 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lhxlr" podUID="ad40b959-18ef-44f8-a81e-d33e19794abf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Nov 03 01:47:20.373888 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:20.373759 5437 patch_prober.go:29] interesting pod/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-8p8gc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Nov 03 01:47:20.373888 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:20.373821 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-8p8gc" podUID="3a5060a0-9abf-4586-aef4-8d681a13b2aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Nov 03 01:47:20.398787 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:20.398750 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:47:20.745536 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:20.737495 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:47:20.951964 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:20.951927 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:47:21.223681 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:21.223639 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:47:21.312624 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:21.312582 5437 patch_prober.go:29] interesting pod/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lhxlr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Nov 03 01:47:21.312839 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:21.312656 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lhxlr" podUID="ad40b959-18ef-44f8-a81e-d33e19794abf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Nov 03 01:47:21.383963 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:21.383912 5437 patch_prober.go:29] interesting pod/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-8p8gc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Nov 03 01:47:21.384182 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:21.384014 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-8p8gc" podUID="3a5060a0-9abf-4586-aef4-8d681a13b2aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Nov 03 01:47:22.173445 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:22.173405 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:47:22.317205 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:22.317172 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lhxlr" Nov 03 01:47:22.317532 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:22.317507 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lhxlr" Nov 03 01:47:22.318167 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:22.318143 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lhxlr" Nov 03 01:47:22.369073 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:22.369041 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-8p8gc" Nov 03 01:47:22.369427 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:22.369359 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-8p8gc" Nov 03 01:47:22.370197 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:22.370181 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-8p8gc" Nov 03 01:47:23.398625 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:23.398588 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:47:25.273670 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:25.273630 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-4952b1-ccfb5b59f-4r2rj" podUID="6faa241e-9510-45af-9e59-e3aa35524217" containerName="payload-processing" containerID="cri-o://9f70104848ce70798044dcf6a223e6d420153c7a815ba11c6ae036766afdebf8" gracePeriod=25 Nov 03 01:47:25.277195 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:25.277151 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:47:25.481623 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:25.481557 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:47:25.561872 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:25.561844 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-4952b1-ccfb5b59f-4r2rj" Nov 03 01:47:25.578052 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:25.578016 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6faa241e-9510-45af-9e59-e3aa35524217-kube-api-access-bg8mx\" (UniqueName: \"kubernetes.io/projected/6faa241e-9510-45af-9e59-e3aa35524217-kube-api-access-bg8mx\") pod \"6faa241e-9510-45af-9e59-e3aa35524217\" (UID: \"6faa241e-9510-45af-9e59-e3aa35524217\") " Nov 03 01:47:25.578185 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:25.578139 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6faa241e-9510-45af-9e59-e3aa35524217-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6faa241e-9510-45af-9e59-e3aa35524217-plugins-config-volume\") pod \"6faa241e-9510-45af-9e59-e3aa35524217\" (UID: \"6faa241e-9510-45af-9e59-e3aa35524217\") " Nov 03 01:47:25.578563 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:25.578534 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6faa241e-9510-45af-9e59-e3aa35524217-plugins-config-volume" pod "6faa241e-9510-45af-9e59-e3aa35524217" (UID: "6faa241e-9510-45af-9e59-e3aa35524217"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:47:25.580223 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:25.580205 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6faa241e-9510-45af-9e59-e3aa35524217-kube-api-access-bg8mx" pod "6faa241e-9510-45af-9e59-e3aa35524217" (UID: "6faa241e-9510-45af-9e59-e3aa35524217"). InnerVolumeSpecName "kube-api-access-bg8mx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:47:25.680015 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:25.679977 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bg8mx\" (UniqueName: \"kubernetes.io/projected/6faa241e-9510-45af-9e59-e3aa35524217-kube-api-access-bg8mx\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:47:25.680015 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:25.680008 5437 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6faa241e-9510-45af-9e59-e3aa35524217-plugins-config-volume\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:47:25.828569 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:25.828535 5437 generic.go:356] "Generic (PLEG): container finished" podID="6faa241e-9510-45af-9e59-e3aa35524217" containerID="9f70104848ce70798044dcf6a223e6d420153c7a815ba11c6ae036766afdebf8" exitCode=0 Nov 03 01:47:25.828747 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:25.828583 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-4952b1-ccfb5b59f-4r2rj" Nov 03 01:47:25.828747 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:25.828618 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-4952b1-ccfb5b59f-4r2rj" event={"ID":"6faa241e-9510-45af-9e59-e3aa35524217","Type":"ContainerDied","Data":"9f70104848ce70798044dcf6a223e6d420153c7a815ba11c6ae036766afdebf8"} Nov 03 01:47:25.828747 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:25.828673 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-4952b1-ccfb5b59f-4r2rj" event={"ID":"6faa241e-9510-45af-9e59-e3aa35524217","Type":"ContainerDied","Data":"782b07eaa4d43eda95fe4dda1c206b299adc6cf4e803fb90fc2f11f611ae4dea"} Nov 03 01:47:25.828747 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:25.828695 5437 scope.go:122] "RemoveContainer" containerID="9f70104848ce70798044dcf6a223e6d420153c7a815ba11c6ae036766afdebf8" Nov 03 01:47:25.839750 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:25.839731 5437 scope.go:122] "RemoveContainer" containerID="9f70104848ce70798044dcf6a223e6d420153c7a815ba11c6ae036766afdebf8" Nov 03 01:47:25.840006 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:47:25.839985 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f70104848ce70798044dcf6a223e6d420153c7a815ba11c6ae036766afdebf8\": container with ID starting with 9f70104848ce70798044dcf6a223e6d420153c7a815ba11c6ae036766afdebf8 not found: ID does not exist" containerID="9f70104848ce70798044dcf6a223e6d420153c7a815ba11c6ae036766afdebf8" Nov 03 01:47:25.840056 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:25.840019 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f70104848ce70798044dcf6a223e6d420153c7a815ba11c6ae036766afdebf8"} err="failed to get container status \"9f70104848ce70798044dcf6a223e6d420153c7a815ba11c6ae036766afdebf8\": rpc error: code = NotFound desc = could not find container \"9f70104848ce70798044dcf6a223e6d420153c7a815ba11c6ae036766afdebf8\": container with ID starting with 9f70104848ce70798044dcf6a223e6d420153c7a815ba11c6ae036766afdebf8 not found: ID does not exist" Nov 03 01:47:25.999674 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:25.999639 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4952b1-ccfb5b59f-4r2rj"] Nov 03 01:47:26.002671 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:26.002642 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4952b1-ccfb5b59f-4r2rj"] Nov 03 01:47:26.090027 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:26.089945 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6faa241e-9510-45af-9e59-e3aa35524217" path="/var/lib/kubelet/pods/6faa241e-9510-45af-9e59-e3aa35524217/volumes" Nov 03 01:47:27.975174 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:27.975135 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:47:28.201409 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:28.201350 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:47:28.406987 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:28.406948 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:47:28.613311 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:28.612598 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:47:29.782504 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:29.782446 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:47:33.985465 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:33.985412 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5e048b-764bb84787-mbchq"] Nov 03 01:47:33.986059 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:33.985701 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5e048b-764bb84787-mbchq" podUID="b8ce6d38-37d9-45ef-a6c0-4f7ad8733998" containerName="maas-api" containerID="cri-o://4d71551d6bbb8b51a1ee0b568e7ab2d7e7efb8fd93963b2d61fae7d703852812" gracePeriod=30 Nov 03 01:47:34.041169 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:34.041078 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:47:34.264429 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:34.259840 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:47:34.300508 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:34.300479 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5e048b-764bb84787-mbchq" Nov 03 01:47:34.420773 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:34.420442 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8ce6d38-37d9-45ef-a6c0-4f7ad8733998-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8ce6d38-37d9-45ef-a6c0-4f7ad8733998-metrics-certs\") pod \"b8ce6d38-37d9-45ef-a6c0-4f7ad8733998\" (UID: \"b8ce6d38-37d9-45ef-a6c0-4f7ad8733998\") " Nov 03 01:47:34.420773 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:34.420696 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8ce6d38-37d9-45ef-a6c0-4f7ad8733998-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8ce6d38-37d9-45ef-a6c0-4f7ad8733998-maas-api-tls\") pod \"b8ce6d38-37d9-45ef-a6c0-4f7ad8733998\" (UID: \"b8ce6d38-37d9-45ef-a6c0-4f7ad8733998\") " Nov 03 01:47:34.422046 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:34.421171 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8ce6d38-37d9-45ef-a6c0-4f7ad8733998-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8ce6d38-37d9-45ef-a6c0-4f7ad8733998-openshift-service-ca\") pod \"b8ce6d38-37d9-45ef-a6c0-4f7ad8733998\" (UID: \"b8ce6d38-37d9-45ef-a6c0-4f7ad8733998\") " Nov 03 01:47:34.422046 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:34.421328 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8ce6d38-37d9-45ef-a6c0-4f7ad8733998-kube-api-access-wzzxq\" (UniqueName: \"kubernetes.io/projected/b8ce6d38-37d9-45ef-a6c0-4f7ad8733998-kube-api-access-wzzxq\") pod \"b8ce6d38-37d9-45ef-a6c0-4f7ad8733998\" (UID: \"b8ce6d38-37d9-45ef-a6c0-4f7ad8733998\") " Nov 03 01:47:34.435418 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:34.426450 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8ce6d38-37d9-45ef-a6c0-4f7ad8733998-metrics-certs" pod "b8ce6d38-37d9-45ef-a6c0-4f7ad8733998" (UID: "b8ce6d38-37d9-45ef-a6c0-4f7ad8733998"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:47:34.435418 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:34.427909 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8ce6d38-37d9-45ef-a6c0-4f7ad8733998-openshift-service-ca" pod "b8ce6d38-37d9-45ef-a6c0-4f7ad8733998" (UID: "b8ce6d38-37d9-45ef-a6c0-4f7ad8733998"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:47:34.449179 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:34.449148 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8ce6d38-37d9-45ef-a6c0-4f7ad8733998-maas-api-tls" pod "b8ce6d38-37d9-45ef-a6c0-4f7ad8733998" (UID: "b8ce6d38-37d9-45ef-a6c0-4f7ad8733998"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:47:34.460289 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:34.460085 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8ce6d38-37d9-45ef-a6c0-4f7ad8733998-kube-api-access-wzzxq" pod "b8ce6d38-37d9-45ef-a6c0-4f7ad8733998" (UID: "b8ce6d38-37d9-45ef-a6c0-4f7ad8733998"). InnerVolumeSpecName "kube-api-access-wzzxq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:47:34.523853 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:34.523760 5437 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8ce6d38-37d9-45ef-a6c0-4f7ad8733998-metrics-certs\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:47:34.523853 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:34.523795 5437 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8ce6d38-37d9-45ef-a6c0-4f7ad8733998-maas-api-tls\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:47:34.523853 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:34.523806 5437 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8ce6d38-37d9-45ef-a6c0-4f7ad8733998-openshift-service-ca\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:47:34.523853 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:34.523815 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzzxq\" (UniqueName: \"kubernetes.io/projected/b8ce6d38-37d9-45ef-a6c0-4f7ad8733998-kube-api-access-wzzxq\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:47:34.888554 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:34.888517 5437 generic.go:356] "Generic (PLEG): container finished" podID="b8ce6d38-37d9-45ef-a6c0-4f7ad8733998" containerID="4d71551d6bbb8b51a1ee0b568e7ab2d7e7efb8fd93963b2d61fae7d703852812" exitCode=0 Nov 03 01:47:34.888739 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:34.888572 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5e048b-764bb84787-mbchq" Nov 03 01:47:34.888739 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:34.888597 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5e048b-764bb84787-mbchq" event={"ID":"b8ce6d38-37d9-45ef-a6c0-4f7ad8733998","Type":"ContainerDied","Data":"4d71551d6bbb8b51a1ee0b568e7ab2d7e7efb8fd93963b2d61fae7d703852812"} Nov 03 01:47:34.888739 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:34.888683 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5e048b-764bb84787-mbchq" event={"ID":"b8ce6d38-37d9-45ef-a6c0-4f7ad8733998","Type":"ContainerDied","Data":"745071cfa17021c505a83ba3449479dd6f1eb3e4e4e9528bf5284e1a5458a1f6"} Nov 03 01:47:34.888739 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:34.888706 5437 scope.go:122] "RemoveContainer" containerID="4d71551d6bbb8b51a1ee0b568e7ab2d7e7efb8fd93963b2d61fae7d703852812" Nov 03 01:47:34.901569 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:34.901551 5437 scope.go:122] "RemoveContainer" containerID="4d71551d6bbb8b51a1ee0b568e7ab2d7e7efb8fd93963b2d61fae7d703852812" Nov 03 01:47:34.901945 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:47:34.901919 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d71551d6bbb8b51a1ee0b568e7ab2d7e7efb8fd93963b2d61fae7d703852812\": container with ID starting with 4d71551d6bbb8b51a1ee0b568e7ab2d7e7efb8fd93963b2d61fae7d703852812 not found: ID does not exist" containerID="4d71551d6bbb8b51a1ee0b568e7ab2d7e7efb8fd93963b2d61fae7d703852812" Nov 03 01:47:34.902055 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:34.901952 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d71551d6bbb8b51a1ee0b568e7ab2d7e7efb8fd93963b2d61fae7d703852812"} err="failed to get container status \"4d71551d6bbb8b51a1ee0b568e7ab2d7e7efb8fd93963b2d61fae7d703852812\": rpc error: code = NotFound desc = could not find container \"4d71551d6bbb8b51a1ee0b568e7ab2d7e7efb8fd93963b2d61fae7d703852812\": container with ID starting with 4d71551d6bbb8b51a1ee0b568e7ab2d7e7efb8fd93963b2d61fae7d703852812 not found: ID does not exist" Nov 03 01:47:35.059921 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:35.059883 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5e048b-764bb84787-mbchq"] Nov 03 01:47:35.111755 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:35.111721 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5e048b-764bb84787-mbchq"] Nov 03 01:47:35.520134 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:35.520099 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:47:36.035800 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:36.035765 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:47:36.141239 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:36.141053 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8ce6d38-37d9-45ef-a6c0-4f7ad8733998" path="/var/lib/kubelet/pods/b8ce6d38-37d9-45ef-a6c0-4f7ad8733998/volumes" Nov 03 01:47:36.252251 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:36.252211 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:47:36.968590 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:36.968552 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:47:37.176511 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:37.176476 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:47:37.619398 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:37.612958 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:47:38.007390 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:38.007279 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:47:38.628085 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:38.628048 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:47:39.137990 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:39.137947 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:47:39.896473 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:39.896438 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:47:40.257230 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:40.257126 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:47:41.316181 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:41.316144 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:47:42.581905 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:42.581863 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:47:42.789825 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:42.789774 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:47:44.036086 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:47:44.036044 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 01:47:45.309040 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:45.308997 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:47:45.542645 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:45.542606 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:47:45.791078 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:45.790986 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:47:46.072078 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:46.072030 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:47:46.943561 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:46.943523 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-118857-5db4c99879-84r9l"] Nov 03 01:47:48.355140 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:48.355084 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:47:51.485090 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:51.485044 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:47:51.944012 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:51.943939 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-118857-5db4c99879-84r9l" podUID="a3547027-5292-4db3-8d5e-fa7e0f792d8f" containerName="payload-pre-processing" containerID="cri-o://b3325a09dd9703ee04da42cf6131460c735280b9a971d12e55f659b916a3b139" gracePeriod=25 Nov 03 01:47:52.098407 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:52.098301 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:47:52.244154 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:52.244120 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-118857-5db4c99879-84r9l" Nov 03 01:47:52.314974 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:52.314934 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3547027-5292-4db3-8d5e-fa7e0f792d8f-kube-api-access-crvt4\" (UniqueName: \"kubernetes.io/projected/a3547027-5292-4db3-8d5e-fa7e0f792d8f-kube-api-access-crvt4\") pod \"a3547027-5292-4db3-8d5e-fa7e0f792d8f\" (UID: \"a3547027-5292-4db3-8d5e-fa7e0f792d8f\") " Nov 03 01:47:52.315216 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:52.315030 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3547027-5292-4db3-8d5e-fa7e0f792d8f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3547027-5292-4db3-8d5e-fa7e0f792d8f-plugins-config-volume\") pod \"a3547027-5292-4db3-8d5e-fa7e0f792d8f\" (UID: \"a3547027-5292-4db3-8d5e-fa7e0f792d8f\") " Nov 03 01:47:52.315454 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:52.315429 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3547027-5292-4db3-8d5e-fa7e0f792d8f-plugins-config-volume" pod "a3547027-5292-4db3-8d5e-fa7e0f792d8f" (UID: "a3547027-5292-4db3-8d5e-fa7e0f792d8f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:47:52.315569 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:52.315555 5437 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3547027-5292-4db3-8d5e-fa7e0f792d8f-plugins-config-volume\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:47:52.317437 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:52.317341 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3547027-5292-4db3-8d5e-fa7e0f792d8f-kube-api-access-crvt4" pod "a3547027-5292-4db3-8d5e-fa7e0f792d8f" (UID: "a3547027-5292-4db3-8d5e-fa7e0f792d8f"). InnerVolumeSpecName "kube-api-access-crvt4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:47:52.417200 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:52.417161 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crvt4\" (UniqueName: \"kubernetes.io/projected/a3547027-5292-4db3-8d5e-fa7e0f792d8f-kube-api-access-crvt4\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:47:52.437566 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:52.437528 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:47:53.019013 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:53.018973 5437 generic.go:356] "Generic (PLEG): container finished" podID="a3547027-5292-4db3-8d5e-fa7e0f792d8f" containerID="b3325a09dd9703ee04da42cf6131460c735280b9a971d12e55f659b916a3b139" exitCode=0 Nov 03 01:47:53.019545 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:53.019019 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-118857-5db4c99879-84r9l" Nov 03 01:47:53.019545 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:53.019026 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-118857-5db4c99879-84r9l" event={"ID":"a3547027-5292-4db3-8d5e-fa7e0f792d8f","Type":"ContainerDied","Data":"b3325a09dd9703ee04da42cf6131460c735280b9a971d12e55f659b916a3b139"} Nov 03 01:47:53.019545 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:53.019081 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-118857-5db4c99879-84r9l" event={"ID":"a3547027-5292-4db3-8d5e-fa7e0f792d8f","Type":"ContainerDied","Data":"c6db455e3bd9fa9c15486cd418b92267b720a7b5f352a502e579d0781fbf9afc"} Nov 03 01:47:53.019545 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:53.019112 5437 scope.go:122] "RemoveContainer" containerID="b3325a09dd9703ee04da42cf6131460c735280b9a971d12e55f659b916a3b139" Nov 03 01:47:53.031097 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:53.031075 5437 scope.go:122] "RemoveContainer" containerID="b3325a09dd9703ee04da42cf6131460c735280b9a971d12e55f659b916a3b139" Nov 03 01:47:53.031447 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:47:53.031424 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3325a09dd9703ee04da42cf6131460c735280b9a971d12e55f659b916a3b139\": container with ID starting with b3325a09dd9703ee04da42cf6131460c735280b9a971d12e55f659b916a3b139 not found: ID does not exist" containerID="b3325a09dd9703ee04da42cf6131460c735280b9a971d12e55f659b916a3b139" Nov 03 01:47:53.031495 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:53.031459 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3325a09dd9703ee04da42cf6131460c735280b9a971d12e55f659b916a3b139"} err="failed to get container status \"b3325a09dd9703ee04da42cf6131460c735280b9a971d12e55f659b916a3b139\": rpc error: code = NotFound desc = could not find container \"b3325a09dd9703ee04da42cf6131460c735280b9a971d12e55f659b916a3b139\": container with ID starting with b3325a09dd9703ee04da42cf6131460c735280b9a971d12e55f659b916a3b139 not found: ID does not exist" Nov 03 01:47:53.190160 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:53.190120 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-118857-5db4c99879-84r9l"] Nov 03 01:47:53.241778 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:53.241736 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-118857-5db4c99879-84r9l"] Nov 03 01:47:53.329806 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:53.329770 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:47:54.089888 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:54.089854 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3547027-5292-4db3-8d5e-fa7e0f792d8f" path="/var/lib/kubelet/pods/a3547027-5292-4db3-8d5e-fa7e0f792d8f/volumes" Nov 03 01:47:56.620130 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:56.620093 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:47:59.358846 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:47:59.358804 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:48:00.463083 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:00.463039 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:48:00.673433 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:00.673397 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:48:01.688249 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:01.688210 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:48:04.999597 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:04.999561 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:48:06.671618 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:06.671507 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:48:08.107521 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:08.107486 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:48:10.765703 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:10.765591 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:48:10.972802 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:10.972764 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:48:11.607357 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:11.607313 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:48:11.864251 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:11.864100 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:48:12.071817 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:12.071779 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:48:14.037606 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:48:14.037568 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 01:48:16.778808 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:16.778759 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:48:16.982236 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:16.982201 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:48:17.182048 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:17.182009 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:48:21.663972 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:21.663930 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:48:25.755479 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:25.755441 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f4e704-7dbf54c85-hvn9h"] Nov 03 01:48:25.755993 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:25.755675 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f4e704-7dbf54c85-hvn9h" podUID="87a89600-532d-4c7a-99f4-a9d41a58bfc5" containerName="maas-api" containerID="cri-o://dc6c078c7976366764c3a13b55f1d21b84fe511433d672a9fe39f5a119af8537" gracePeriod=30 Nov 03 01:48:25.909799 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:25.909761 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:48:26.062295 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:26.062263 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f4e704-7dbf54c85-hvn9h" Nov 03 01:48:26.219940 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:26.219894 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/87a89600-532d-4c7a-99f4-a9d41a58bfc5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/87a89600-532d-4c7a-99f4-a9d41a58bfc5-metrics-certs\") pod \"87a89600-532d-4c7a-99f4-a9d41a58bfc5\" (UID: \"87a89600-532d-4c7a-99f4-a9d41a58bfc5\") " Nov 03 01:48:26.220213 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:26.219957 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/87a89600-532d-4c7a-99f4-a9d41a58bfc5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87a89600-532d-4c7a-99f4-a9d41a58bfc5-maas-api-tls\") pod \"87a89600-532d-4c7a-99f4-a9d41a58bfc5\" (UID: \"87a89600-532d-4c7a-99f4-a9d41a58bfc5\") " Nov 03 01:48:26.220213 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:26.220038 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87a89600-532d-4c7a-99f4-a9d41a58bfc5-kube-api-access-dhfmr\" (UniqueName: \"kubernetes.io/projected/87a89600-532d-4c7a-99f4-a9d41a58bfc5-kube-api-access-dhfmr\") pod \"87a89600-532d-4c7a-99f4-a9d41a58bfc5\" (UID: \"87a89600-532d-4c7a-99f4-a9d41a58bfc5\") " Nov 03 01:48:26.220213 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:26.220161 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87a89600-532d-4c7a-99f4-a9d41a58bfc5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87a89600-532d-4c7a-99f4-a9d41a58bfc5-openshift-service-ca\") pod \"87a89600-532d-4c7a-99f4-a9d41a58bfc5\" (UID: \"87a89600-532d-4c7a-99f4-a9d41a58bfc5\") " Nov 03 01:48:26.220857 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:26.220817 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87a89600-532d-4c7a-99f4-a9d41a58bfc5-openshift-service-ca" pod "87a89600-532d-4c7a-99f4-a9d41a58bfc5" (UID: "87a89600-532d-4c7a-99f4-a9d41a58bfc5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:48:26.225244 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:26.225117 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87a89600-532d-4c7a-99f4-a9d41a58bfc5-metrics-certs" pod "87a89600-532d-4c7a-99f4-a9d41a58bfc5" (UID: "87a89600-532d-4c7a-99f4-a9d41a58bfc5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:48:26.225244 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:26.225132 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87a89600-532d-4c7a-99f4-a9d41a58bfc5-kube-api-access-dhfmr" pod "87a89600-532d-4c7a-99f4-a9d41a58bfc5" (UID: "87a89600-532d-4c7a-99f4-a9d41a58bfc5"). InnerVolumeSpecName "kube-api-access-dhfmr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:48:26.226874 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:26.226838 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87a89600-532d-4c7a-99f4-a9d41a58bfc5-maas-api-tls" pod "87a89600-532d-4c7a-99f4-a9d41a58bfc5" (UID: "87a89600-532d-4c7a-99f4-a9d41a58bfc5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:48:26.282642 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:26.282589 5437 generic.go:356] "Generic (PLEG): container finished" podID="87a89600-532d-4c7a-99f4-a9d41a58bfc5" containerID="dc6c078c7976366764c3a13b55f1d21b84fe511433d672a9fe39f5a119af8537" exitCode=0 Nov 03 01:48:26.282843 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:26.282644 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f4e704-7dbf54c85-hvn9h" Nov 03 01:48:26.282843 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:26.282671 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f4e704-7dbf54c85-hvn9h" event={"ID":"87a89600-532d-4c7a-99f4-a9d41a58bfc5","Type":"ContainerDied","Data":"dc6c078c7976366764c3a13b55f1d21b84fe511433d672a9fe39f5a119af8537"} Nov 03 01:48:26.282843 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:26.282729 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f4e704-7dbf54c85-hvn9h" event={"ID":"87a89600-532d-4c7a-99f4-a9d41a58bfc5","Type":"ContainerDied","Data":"c776aa502e6e0d6e01d22d2d43a4aca94ba10061cc6f751d565593ee01c89c30"} Nov 03 01:48:26.282843 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:26.282752 5437 scope.go:122] "RemoveContainer" containerID="dc6c078c7976366764c3a13b55f1d21b84fe511433d672a9fe39f5a119af8537" Nov 03 01:48:26.293568 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:26.293544 5437 scope.go:122] "RemoveContainer" containerID="dc6c078c7976366764c3a13b55f1d21b84fe511433d672a9fe39f5a119af8537" Nov 03 01:48:26.293907 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:48:26.293884 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc6c078c7976366764c3a13b55f1d21b84fe511433d672a9fe39f5a119af8537\": container with ID starting with dc6c078c7976366764c3a13b55f1d21b84fe511433d672a9fe39f5a119af8537 not found: ID does not exist" containerID="dc6c078c7976366764c3a13b55f1d21b84fe511433d672a9fe39f5a119af8537" Nov 03 01:48:26.293959 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:26.293919 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc6c078c7976366764c3a13b55f1d21b84fe511433d672a9fe39f5a119af8537"} err="failed to get container status \"dc6c078c7976366764c3a13b55f1d21b84fe511433d672a9fe39f5a119af8537\": rpc error: code = NotFound desc = could not find container \"dc6c078c7976366764c3a13b55f1d21b84fe511433d672a9fe39f5a119af8537\": container with ID starting with dc6c078c7976366764c3a13b55f1d21b84fe511433d672a9fe39f5a119af8537 not found: ID does not exist" Nov 03 01:48:26.326090 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:26.326042 5437 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87a89600-532d-4c7a-99f4-a9d41a58bfc5-metrics-certs\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:48:26.326090 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:26.326083 5437 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87a89600-532d-4c7a-99f4-a9d41a58bfc5-maas-api-tls\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:48:26.326400 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:26.326103 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhfmr\" (UniqueName: \"kubernetes.io/projected/87a89600-532d-4c7a-99f4-a9d41a58bfc5-kube-api-access-dhfmr\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:48:26.326400 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:26.326118 5437 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87a89600-532d-4c7a-99f4-a9d41a58bfc5-openshift-service-ca\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:48:26.460597 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:26.460555 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f4e704-7dbf54c85-hvn9h"] Nov 03 01:48:26.463593 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:26.463565 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f4e704-7dbf54c85-hvn9h"] Nov 03 01:48:27.528583 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:27.528501 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:48:27.737157 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:27.737113 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:48:27.948234 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:27.948189 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:48:28.091577 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:28.091540 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87a89600-532d-4c7a-99f4-a9d41a58bfc5" path="/var/lib/kubelet/pods/87a89600-532d-4c7a-99f4-a9d41a58bfc5/volumes" Nov 03 01:48:29.996265 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:29.996230 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:48:30.200313 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:30.200272 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:48:31.545244 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:31.545200 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:48:31.986169 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:31.986133 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-pm59c"] Nov 03 01:48:31.986418 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:31.986395 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-pm59c" podUID="5b441095-b0c0-4bbe-89af-5cf30727a4a7" containerName="istio-proxy" containerID="cri-o://689cceff9761fcc394d4b801a557ed706bb8bd78ceb053ef08185016c9898043" gracePeriod=30 Nov 03 01:48:32.014061 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:32.014022 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-cvlqf"] Nov 03 01:48:32.014264 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:32.014241 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-cvlqf" podUID="ea981b45-743a-4e06-83c9-519fbda3d7d8" containerName="istio-proxy" containerID="cri-o://2901cf672fe080b882e623aea3d6c30aacf3f3ed81c901e8186e1f49c085b123" gracePeriod=30 Nov 03 01:48:32.041797 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:32.041760 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:48:32.712977 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:32.712940 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:48:32.919867 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:32.919830 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:48:34.577077 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:34.577040 5437 patch_prober.go:29] interesting pod/e2e-api-a-f4e704-openshift-default-576d67d949-cvlqf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 01:48:34.577483 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:34.577102 5437 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-cvlqf" podUID="ea981b45-743a-4e06-83c9-519fbda3d7d8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 01:48:35.263497 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:35.263461 5437 patch_prober.go:29] interesting pod/e2e-api-a-f4e704-openshift-default-576d67d949-pm59c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 01:48:35.263668 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:35.263534 5437 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-pm59c" podUID="5b441095-b0c0-4bbe-89af-5cf30727a4a7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 01:48:37.237338 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:37.237248 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:48:37.312184 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:37.312159 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-pm59c" Nov 03 01:48:37.368185 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:37.368144 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-cvlqf" Nov 03 01:48:37.375405 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:37.375321 5437 generic.go:356] "Generic (PLEG): container finished" podID="ea981b45-743a-4e06-83c9-519fbda3d7d8" containerID="2901cf672fe080b882e623aea3d6c30aacf3f3ed81c901e8186e1f49c085b123" exitCode=0 Nov 03 01:48:37.375593 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:37.375582 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-cvlqf" Nov 03 01:48:37.375721 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:37.375699 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-cvlqf" event={"ID":"ea981b45-743a-4e06-83c9-519fbda3d7d8","Type":"ContainerDied","Data":"2901cf672fe080b882e623aea3d6c30aacf3f3ed81c901e8186e1f49c085b123"} Nov 03 01:48:37.375789 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:37.375745 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-cvlqf" event={"ID":"ea981b45-743a-4e06-83c9-519fbda3d7d8","Type":"ContainerDied","Data":"161527535e07fbfb420154a31d4303afb152c5d9682e416103f052258ab1dff5"} Nov 03 01:48:37.375789 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:37.375768 5437 scope.go:122] "RemoveContainer" containerID="2901cf672fe080b882e623aea3d6c30aacf3f3ed81c901e8186e1f49c085b123" Nov 03 01:48:37.380248 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:37.380204 5437 generic.go:356] "Generic (PLEG): container finished" podID="5b441095-b0c0-4bbe-89af-5cf30727a4a7" containerID="689cceff9761fcc394d4b801a557ed706bb8bd78ceb053ef08185016c9898043" exitCode=0 Nov 03 01:48:37.380583 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:37.380340 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-pm59c" Nov 03 01:48:37.380727 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:37.380376 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-pm59c" event={"ID":"5b441095-b0c0-4bbe-89af-5cf30727a4a7","Type":"ContainerDied","Data":"689cceff9761fcc394d4b801a557ed706bb8bd78ceb053ef08185016c9898043"} Nov 03 01:48:37.380808 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:37.380762 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-pm59c" event={"ID":"5b441095-b0c0-4bbe-89af-5cf30727a4a7","Type":"ContainerDied","Data":"c28f38c17c4b176b8a3b2a79c539e930a00b23e1b2801e32253cbbdc71d19c42"} Nov 03 01:48:37.386580 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:37.386555 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b441095-b0c0-4bbe-89af-5cf30727a4a7-istio-token\" (UniqueName: \"kubernetes.io/projected/5b441095-b0c0-4bbe-89af-5cf30727a4a7-istio-token\") pod \"5b441095-b0c0-4bbe-89af-5cf30727a4a7\" (UID: \"5b441095-b0c0-4bbe-89af-5cf30727a4a7\") " Nov 03 01:48:37.386709 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:37.386610 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b441095-b0c0-4bbe-89af-5cf30727a4a7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b441095-b0c0-4bbe-89af-5cf30727a4a7-istio-envoy\") pod \"5b441095-b0c0-4bbe-89af-5cf30727a4a7\" (UID: \"5b441095-b0c0-4bbe-89af-5cf30727a4a7\") " Nov 03 01:48:37.386779 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:37.386755 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b441095-b0c0-4bbe-89af-5cf30727a4a7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b441095-b0c0-4bbe-89af-5cf30727a4a7-istiod-ca-cert\") pod \"5b441095-b0c0-4bbe-89af-5cf30727a4a7\" (UID: \"5b441095-b0c0-4bbe-89af-5cf30727a4a7\") " Nov 03 01:48:37.386836 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:37.386797 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b441095-b0c0-4bbe-89af-5cf30727a4a7-kube-api-access-6pfh9\" (UniqueName: \"kubernetes.io/projected/5b441095-b0c0-4bbe-89af-5cf30727a4a7-kube-api-access-6pfh9\") pod \"5b441095-b0c0-4bbe-89af-5cf30727a4a7\" (UID: \"5b441095-b0c0-4bbe-89af-5cf30727a4a7\") " Nov 03 01:48:37.386899 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:37.386849 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b441095-b0c0-4bbe-89af-5cf30727a4a7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b441095-b0c0-4bbe-89af-5cf30727a4a7-workload-certs\") pod \"5b441095-b0c0-4bbe-89af-5cf30727a4a7\" (UID: \"5b441095-b0c0-4bbe-89af-5cf30727a4a7\") " Nov 03 01:48:37.386899 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:37.386886 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5b441095-b0c0-4bbe-89af-5cf30727a4a7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b441095-b0c0-4bbe-89af-5cf30727a4a7-istio-podinfo\") pod \"5b441095-b0c0-4bbe-89af-5cf30727a4a7\" (UID: \"5b441095-b0c0-4bbe-89af-5cf30727a4a7\") " Nov 03 01:48:37.387005 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:37.386923 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b441095-b0c0-4bbe-89af-5cf30727a4a7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b441095-b0c0-4bbe-89af-5cf30727a4a7-workload-socket\") pod \"5b441095-b0c0-4bbe-89af-5cf30727a4a7\" (UID: \"5b441095-b0c0-4bbe-89af-5cf30727a4a7\") " Nov 03 01:48:37.387005 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:37.386965 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b441095-b0c0-4bbe-89af-5cf30727a4a7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b441095-b0c0-4bbe-89af-5cf30727a4a7-istio-data\") pod \"5b441095-b0c0-4bbe-89af-5cf30727a4a7\" (UID: \"5b441095-b0c0-4bbe-89af-5cf30727a4a7\") " Nov 03 01:48:37.387171 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:37.387066 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b441095-b0c0-4bbe-89af-5cf30727a4a7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b441095-b0c0-4bbe-89af-5cf30727a4a7-credential-socket\") pod \"5b441095-b0c0-4bbe-89af-5cf30727a4a7\" (UID: \"5b441095-b0c0-4bbe-89af-5cf30727a4a7\") " Nov 03 01:48:37.387494 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:37.387461 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b441095-b0c0-4bbe-89af-5cf30727a4a7-workload-socket" pod "5b441095-b0c0-4bbe-89af-5cf30727a4a7" (UID: "5b441095-b0c0-4bbe-89af-5cf30727a4a7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:48:37.388231 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:37.387602 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b441095-b0c0-4bbe-89af-5cf30727a4a7-istiod-ca-cert" pod "5b441095-b0c0-4bbe-89af-5cf30727a4a7" (UID: "5b441095-b0c0-4bbe-89af-5cf30727a4a7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:48:37.388231 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:37.387893 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b441095-b0c0-4bbe-89af-5cf30727a4a7-credential-socket" pod "5b441095-b0c0-4bbe-89af-5cf30727a4a7" (UID: "5b441095-b0c0-4bbe-89af-5cf30727a4a7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:48:37.388231 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:37.388178 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b441095-b0c0-4bbe-89af-5cf30727a4a7-istio-data" pod "5b441095-b0c0-4bbe-89af-5cf30727a4a7" (UID: "5b441095-b0c0-4bbe-89af-5cf30727a4a7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:48:37.388231 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:37.388186 5437 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b441095-b0c0-4bbe-89af-5cf30727a4a7-istiod-ca-cert\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:48:37.388561 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:37.388246 5437 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b441095-b0c0-4bbe-89af-5cf30727a4a7-workload-socket\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:48:37.388561 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:37.388265 5437 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b441095-b0c0-4bbe-89af-5cf30727a4a7-credential-socket\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:48:37.388561 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:37.388328 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b441095-b0c0-4bbe-89af-5cf30727a4a7-workload-certs" pod "5b441095-b0c0-4bbe-89af-5cf30727a4a7" (UID: "5b441095-b0c0-4bbe-89af-5cf30727a4a7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:48:37.391931 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:37.391886 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b441095-b0c0-4bbe-89af-5cf30727a4a7-istio-envoy" pod "5b441095-b0c0-4bbe-89af-5cf30727a4a7" (UID: "5b441095-b0c0-4bbe-89af-5cf30727a4a7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:48:37.391931 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:37.391912 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b441095-b0c0-4bbe-89af-5cf30727a4a7-istio-token" pod "5b441095-b0c0-4bbe-89af-5cf30727a4a7" (UID: "5b441095-b0c0-4bbe-89af-5cf30727a4a7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:48:37.392075 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:37.392034 5437 scope.go:122] "RemoveContainer" containerID="2901cf672fe080b882e623aea3d6c30aacf3f3ed81c901e8186e1f49c085b123" Nov 03 01:48:37.392172 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:37.392143 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5b441095-b0c0-4bbe-89af-5cf30727a4a7-istio-podinfo" pod "5b441095-b0c0-4bbe-89af-5cf30727a4a7" (UID: "5b441095-b0c0-4bbe-89af-5cf30727a4a7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:48:37.392508 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:48:37.392474 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2901cf672fe080b882e623aea3d6c30aacf3f3ed81c901e8186e1f49c085b123\": container with ID starting with 2901cf672fe080b882e623aea3d6c30aacf3f3ed81c901e8186e1f49c085b123 not found: ID does not exist" containerID="2901cf672fe080b882e623aea3d6c30aacf3f3ed81c901e8186e1f49c085b123" Nov 03 01:48:37.392624 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:37.392514 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2901cf672fe080b882e623aea3d6c30aacf3f3ed81c901e8186e1f49c085b123"} err="failed to get container status \"2901cf672fe080b882e623aea3d6c30aacf3f3ed81c901e8186e1f49c085b123\": rpc error: code = NotFound desc = could not find container \"2901cf672fe080b882e623aea3d6c30aacf3f3ed81c901e8186e1f49c085b123\": container with ID starting with 2901cf672fe080b882e623aea3d6c30aacf3f3ed81c901e8186e1f49c085b123 not found: ID does not exist" Nov 03 01:48:37.392624 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:37.392539 5437 scope.go:122] "RemoveContainer" containerID="689cceff9761fcc394d4b801a557ed706bb8bd78ceb053ef08185016c9898043" Nov 03 01:48:37.397587 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:37.394870 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b441095-b0c0-4bbe-89af-5cf30727a4a7-kube-api-access-6pfh9" pod "5b441095-b0c0-4bbe-89af-5cf30727a4a7" (UID: "5b441095-b0c0-4bbe-89af-5cf30727a4a7"). InnerVolumeSpecName "kube-api-access-6pfh9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:48:37.406574 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:37.406542 5437 scope.go:122] "RemoveContainer" containerID="689cceff9761fcc394d4b801a557ed706bb8bd78ceb053ef08185016c9898043" Nov 03 01:48:37.406959 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:48:37.406927 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"689cceff9761fcc394d4b801a557ed706bb8bd78ceb053ef08185016c9898043\": container with ID starting with 689cceff9761fcc394d4b801a557ed706bb8bd78ceb053ef08185016c9898043 not found: ID does not exist" containerID="689cceff9761fcc394d4b801a557ed706bb8bd78ceb053ef08185016c9898043" Nov 03 01:48:37.407071 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:37.406970 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"689cceff9761fcc394d4b801a557ed706bb8bd78ceb053ef08185016c9898043"} err="failed to get container status \"689cceff9761fcc394d4b801a557ed706bb8bd78ceb053ef08185016c9898043\": rpc error: code = NotFound desc = could not find container \"689cceff9761fcc394d4b801a557ed706bb8bd78ceb053ef08185016c9898043\": container with ID starting with 689cceff9761fcc394d4b801a557ed706bb8bd78ceb053ef08185016c9898043 not found: ID does not exist" Nov 03 01:48:37.489290 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:37.489259 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea981b45-743a-4e06-83c9-519fbda3d7d8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea981b45-743a-4e06-83c9-519fbda3d7d8-workload-certs\") pod \"ea981b45-743a-4e06-83c9-519fbda3d7d8\" (UID: \"ea981b45-743a-4e06-83c9-519fbda3d7d8\") " Nov 03 01:48:37.489523 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:37.489313 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea981b45-743a-4e06-83c9-519fbda3d7d8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea981b45-743a-4e06-83c9-519fbda3d7d8-credential-socket\") pod \"ea981b45-743a-4e06-83c9-519fbda3d7d8\" (UID: \"ea981b45-743a-4e06-83c9-519fbda3d7d8\") " Nov 03 01:48:37.489523 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:37.489431 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea981b45-743a-4e06-83c9-519fbda3d7d8-kube-api-access-2q6f7\" (UniqueName: \"kubernetes.io/projected/ea981b45-743a-4e06-83c9-519fbda3d7d8-kube-api-access-2q6f7\") pod \"ea981b45-743a-4e06-83c9-519fbda3d7d8\" (UID: \"ea981b45-743a-4e06-83c9-519fbda3d7d8\") " Nov 03 01:48:37.489523 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:37.489488 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea981b45-743a-4e06-83c9-519fbda3d7d8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea981b45-743a-4e06-83c9-519fbda3d7d8-workload-socket\") pod \"ea981b45-743a-4e06-83c9-519fbda3d7d8\" (UID: \"ea981b45-743a-4e06-83c9-519fbda3d7d8\") " Nov 03 01:48:37.489710 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:37.489531 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea981b45-743a-4e06-83c9-519fbda3d7d8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea981b45-743a-4e06-83c9-519fbda3d7d8-istio-envoy\") pod \"ea981b45-743a-4e06-83c9-519fbda3d7d8\" (UID: \"ea981b45-743a-4e06-83c9-519fbda3d7d8\") " Nov 03 01:48:37.489710 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:37.489579 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ea981b45-743a-4e06-83c9-519fbda3d7d8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea981b45-743a-4e06-83c9-519fbda3d7d8-istio-podinfo\") pod \"ea981b45-743a-4e06-83c9-519fbda3d7d8\" (UID: \"ea981b45-743a-4e06-83c9-519fbda3d7d8\") " Nov 03 01:48:37.489710 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:37.489601 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea981b45-743a-4e06-83c9-519fbda3d7d8-workload-certs" pod "ea981b45-743a-4e06-83c9-519fbda3d7d8" (UID: "ea981b45-743a-4e06-83c9-519fbda3d7d8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:48:37.489710 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:37.489618 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea981b45-743a-4e06-83c9-519fbda3d7d8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea981b45-743a-4e06-83c9-519fbda3d7d8-istiod-ca-cert\") pod \"ea981b45-743a-4e06-83c9-519fbda3d7d8\" (UID: \"ea981b45-743a-4e06-83c9-519fbda3d7d8\") " Nov 03 01:48:37.489710 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:37.489671 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea981b45-743a-4e06-83c9-519fbda3d7d8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea981b45-743a-4e06-83c9-519fbda3d7d8-istio-data\") pod \"ea981b45-743a-4e06-83c9-519fbda3d7d8\" (UID: \"ea981b45-743a-4e06-83c9-519fbda3d7d8\") " Nov 03 01:48:37.489710 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:37.489710 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea981b45-743a-4e06-83c9-519fbda3d7d8-istio-token\" (UniqueName: \"kubernetes.io/projected/ea981b45-743a-4e06-83c9-519fbda3d7d8-istio-token\") pod \"ea981b45-743a-4e06-83c9-519fbda3d7d8\" (UID: \"ea981b45-743a-4e06-83c9-519fbda3d7d8\") " Nov 03 01:48:37.490007 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:37.489811 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea981b45-743a-4e06-83c9-519fbda3d7d8-workload-socket" pod "ea981b45-743a-4e06-83c9-519fbda3d7d8" (UID: "ea981b45-743a-4e06-83c9-519fbda3d7d8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:48:37.490097 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:37.490076 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea981b45-743a-4e06-83c9-519fbda3d7d8-credential-socket" pod "ea981b45-743a-4e06-83c9-519fbda3d7d8" (UID: "ea981b45-743a-4e06-83c9-519fbda3d7d8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:48:37.490234 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:37.490197 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea981b45-743a-4e06-83c9-519fbda3d7d8-istiod-ca-cert" pod "ea981b45-743a-4e06-83c9-519fbda3d7d8" (UID: "ea981b45-743a-4e06-83c9-519fbda3d7d8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:48:37.490487 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:37.490322 5437 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea981b45-743a-4e06-83c9-519fbda3d7d8-workload-certs\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:48:37.490487 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:37.490402 5437 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea981b45-743a-4e06-83c9-519fbda3d7d8-credential-socket\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:48:37.490487 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:37.490421 5437 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b441095-b0c0-4bbe-89af-5cf30727a4a7-istio-token\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:48:37.490487 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:37.490435 5437 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b441095-b0c0-4bbe-89af-5cf30727a4a7-istio-envoy\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:48:37.490487 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:37.490450 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6pfh9\" (UniqueName: \"kubernetes.io/projected/5b441095-b0c0-4bbe-89af-5cf30727a4a7-kube-api-access-6pfh9\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:48:37.490487 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:37.490464 5437 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b441095-b0c0-4bbe-89af-5cf30727a4a7-workload-certs\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:48:37.490487 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:37.490477 5437 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b441095-b0c0-4bbe-89af-5cf30727a4a7-istio-podinfo\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:48:37.490487 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:37.490474 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea981b45-743a-4e06-83c9-519fbda3d7d8-istio-data" pod "ea981b45-743a-4e06-83c9-519fbda3d7d8" (UID: "ea981b45-743a-4e06-83c9-519fbda3d7d8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:48:37.490487 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:37.490490 5437 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea981b45-743a-4e06-83c9-519fbda3d7d8-workload-socket\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:48:37.490952 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:37.490505 5437 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b441095-b0c0-4bbe-89af-5cf30727a4a7-istio-data\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:48:37.490952 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:37.490519 5437 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea981b45-743a-4e06-83c9-519fbda3d7d8-istiod-ca-cert\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:48:37.492429 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:37.492360 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea981b45-743a-4e06-83c9-519fbda3d7d8-kube-api-access-2q6f7" pod "ea981b45-743a-4e06-83c9-519fbda3d7d8" (UID: "ea981b45-743a-4e06-83c9-519fbda3d7d8"). InnerVolumeSpecName "kube-api-access-2q6f7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:48:37.492530 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:37.492504 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea981b45-743a-4e06-83c9-519fbda3d7d8-istio-envoy" pod "ea981b45-743a-4e06-83c9-519fbda3d7d8" (UID: "ea981b45-743a-4e06-83c9-519fbda3d7d8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:48:37.492659 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:37.492633 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ea981b45-743a-4e06-83c9-519fbda3d7d8-istio-podinfo" pod "ea981b45-743a-4e06-83c9-519fbda3d7d8" (UID: "ea981b45-743a-4e06-83c9-519fbda3d7d8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:48:37.493517 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:37.493495 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea981b45-743a-4e06-83c9-519fbda3d7d8-istio-token" pod "ea981b45-743a-4e06-83c9-519fbda3d7d8" (UID: "ea981b45-743a-4e06-83c9-519fbda3d7d8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:48:37.592212 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:37.592090 5437 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea981b45-743a-4e06-83c9-519fbda3d7d8-istio-data\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:48:37.592212 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:37.592130 5437 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea981b45-743a-4e06-83c9-519fbda3d7d8-istio-token\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:48:37.592212 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:37.592153 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2q6f7\" (UniqueName: \"kubernetes.io/projected/ea981b45-743a-4e06-83c9-519fbda3d7d8-kube-api-access-2q6f7\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:48:37.592212 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:37.592169 5437 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea981b45-743a-4e06-83c9-519fbda3d7d8-istio-envoy\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:48:37.592212 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:37.592183 5437 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea981b45-743a-4e06-83c9-519fbda3d7d8-istio-podinfo\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:48:37.804444 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:37.804405 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-cvlqf"] Nov 03 01:48:37.807924 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:37.807894 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-cvlqf"] Nov 03 01:48:37.923403 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:37.923291 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-pm59c"] Nov 03 01:48:37.927184 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:37.927152 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-pm59c"] Nov 03 01:48:38.097619 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:38.097578 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b441095-b0c0-4bbe-89af-5cf30727a4a7" path="/var/lib/kubelet/pods/5b441095-b0c0-4bbe-89af-5cf30727a4a7/volumes" Nov 03 01:48:38.098632 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:38.098595 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea981b45-743a-4e06-83c9-519fbda3d7d8" path="/var/lib/kubelet/pods/ea981b45-743a-4e06-83c9-519fbda3d7d8/volumes" Nov 03 01:48:39.081798 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:39.081752 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:48:39.285390 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:39.283194 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:48:40.418524 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:40.418481 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:48:40.632294 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:40.632251 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:48:43.169531 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:43.169486 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:48:43.373359 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:43.373311 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:48:44.035732 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:48:44.035687 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 01:48:44.312570 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:44.312530 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:48:44.720015 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:44.719923 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:48:45.206720 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:45.206681 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:48:45.526172 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:45.526075 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:48:45.890977 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:45.890937 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:48:46.488517 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:46.488485 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:48:47.207317 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:47.206971 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:48:47.445031 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:47.444987 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:48:49.527844 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:49.527336 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:48:51.592970 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:51.592930 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:48:51.807948 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:51.807901 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:48:52.012208 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:52.012078 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:48:54.807629 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:54.807578 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:48:55.016423 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:55.016358 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:48:57.285787 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:57.285744 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:48:59.214681 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:48:59.214630 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:49:00.980128 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:00.980088 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:49:05.773652 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:05.773608 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:49:07.493718 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:07.493674 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:49:07.696289 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:07.696250 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:49:09.055645 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:09.055579 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:49:09.515025 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:09.514921 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lhxlr"] Nov 03 01:49:09.515213 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:09.515182 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lhxlr" podUID="ad40b959-18ef-44f8-a81e-d33e19794abf" containerName="istio-proxy" containerID="cri-o://56f673e8057177a93349f206f1d2e80bf649739f9f562e6e72f29bda0ad5ce85" gracePeriod=30 Nov 03 01:49:09.528821 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:09.528780 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lrcmd"] Nov 03 01:49:09.529037 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:09.529009 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lrcmd" podUID="296123df-fd9f-4268-a61c-bb6ee52240be" containerName="istio-proxy" containerID="cri-o://abb436119f52563e452c26ff2e2478d0a16d35cb488a27a84ed15e12f6ddb21c" gracePeriod=30 Nov 03 01:49:09.545455 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:09.545343 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-8p8gc"] Nov 03 01:49:09.545763 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:09.545732 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-8p8gc" podUID="3a5060a0-9abf-4586-aef4-8d681a13b2aa" containerName="istio-proxy" containerID="cri-o://48cf9404fc106d59f6c5cf3c6cac07d0da8ab76c409859f9b5f7f6319666d8dd" gracePeriod=30 Nov 03 01:49:09.560636 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:09.560596 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:49:10.389329 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:10.389287 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:49:10.828133 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:10.828083 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:49:11.061562 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:11.061504 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:49:11.365848 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:11.365812 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:49:12.771168 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:12.771132 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:49:12.978018 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:12.977978 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:49:13.194831 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:13.194787 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:49:14.037039 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:49:14.037001 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 01:49:14.834767 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:14.834734 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lhxlr" Nov 03 01:49:14.867782 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:14.867748 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad40b959-18ef-44f8-a81e-d33e19794abf-istio-token\" (UniqueName: \"kubernetes.io/projected/ad40b959-18ef-44f8-a81e-d33e19794abf-istio-token\") pod \"ad40b959-18ef-44f8-a81e-d33e19794abf\" (UID: \"ad40b959-18ef-44f8-a81e-d33e19794abf\") " Nov 03 01:49:14.867782 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:14.867794 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ad40b959-18ef-44f8-a81e-d33e19794abf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad40b959-18ef-44f8-a81e-d33e19794abf-istio-podinfo\") pod \"ad40b959-18ef-44f8-a81e-d33e19794abf\" (UID: \"ad40b959-18ef-44f8-a81e-d33e19794abf\") " Nov 03 01:49:14.868053 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:14.867896 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad40b959-18ef-44f8-a81e-d33e19794abf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad40b959-18ef-44f8-a81e-d33e19794abf-istiod-ca-cert\") pod \"ad40b959-18ef-44f8-a81e-d33e19794abf\" (UID: \"ad40b959-18ef-44f8-a81e-d33e19794abf\") " Nov 03 01:49:14.868053 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:14.867926 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad40b959-18ef-44f8-a81e-d33e19794abf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad40b959-18ef-44f8-a81e-d33e19794abf-istio-envoy\") pod \"ad40b959-18ef-44f8-a81e-d33e19794abf\" (UID: \"ad40b959-18ef-44f8-a81e-d33e19794abf\") " Nov 03 01:49:14.868053 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:14.867966 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad40b959-18ef-44f8-a81e-d33e19794abf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad40b959-18ef-44f8-a81e-d33e19794abf-credential-socket\") pod \"ad40b959-18ef-44f8-a81e-d33e19794abf\" (UID: \"ad40b959-18ef-44f8-a81e-d33e19794abf\") " Nov 03 01:49:14.868053 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:14.868003 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad40b959-18ef-44f8-a81e-d33e19794abf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad40b959-18ef-44f8-a81e-d33e19794abf-istio-data\") pod \"ad40b959-18ef-44f8-a81e-d33e19794abf\" (UID: \"ad40b959-18ef-44f8-a81e-d33e19794abf\") " Nov 03 01:49:14.868053 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:14.868038 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad40b959-18ef-44f8-a81e-d33e19794abf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad40b959-18ef-44f8-a81e-d33e19794abf-workload-socket\") pod \"ad40b959-18ef-44f8-a81e-d33e19794abf\" (UID: \"ad40b959-18ef-44f8-a81e-d33e19794abf\") " Nov 03 01:49:14.868286 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:14.868203 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad40b959-18ef-44f8-a81e-d33e19794abf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad40b959-18ef-44f8-a81e-d33e19794abf-workload-certs\") pod \"ad40b959-18ef-44f8-a81e-d33e19794abf\" (UID: \"ad40b959-18ef-44f8-a81e-d33e19794abf\") " Nov 03 01:49:14.868286 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:14.868266 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad40b959-18ef-44f8-a81e-d33e19794abf-kube-api-access-qbpvb\" (UniqueName: \"kubernetes.io/projected/ad40b959-18ef-44f8-a81e-d33e19794abf-kube-api-access-qbpvb\") pod \"ad40b959-18ef-44f8-a81e-d33e19794abf\" (UID: \"ad40b959-18ef-44f8-a81e-d33e19794abf\") " Nov 03 01:49:14.868492 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:14.868460 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad40b959-18ef-44f8-a81e-d33e19794abf-workload-socket" pod "ad40b959-18ef-44f8-a81e-d33e19794abf" (UID: "ad40b959-18ef-44f8-a81e-d33e19794abf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:49:14.868912 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:14.868887 5437 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad40b959-18ef-44f8-a81e-d33e19794abf-workload-socket\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:49:14.869187 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:14.869168 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad40b959-18ef-44f8-a81e-d33e19794abf-workload-certs" pod "ad40b959-18ef-44f8-a81e-d33e19794abf" (UID: "ad40b959-18ef-44f8-a81e-d33e19794abf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:49:14.869848 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:14.869818 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad40b959-18ef-44f8-a81e-d33e19794abf-istiod-ca-cert" pod "ad40b959-18ef-44f8-a81e-d33e19794abf" (UID: "ad40b959-18ef-44f8-a81e-d33e19794abf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:49:14.869954 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:14.869899 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad40b959-18ef-44f8-a81e-d33e19794abf-credential-socket" pod "ad40b959-18ef-44f8-a81e-d33e19794abf" (UID: "ad40b959-18ef-44f8-a81e-d33e19794abf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:49:14.870028 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:14.869955 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad40b959-18ef-44f8-a81e-d33e19794abf-istio-data" pod "ad40b959-18ef-44f8-a81e-d33e19794abf" (UID: "ad40b959-18ef-44f8-a81e-d33e19794abf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:49:14.870827 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:14.870800 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad40b959-18ef-44f8-a81e-d33e19794abf-istio-token" pod "ad40b959-18ef-44f8-a81e-d33e19794abf" (UID: "ad40b959-18ef-44f8-a81e-d33e19794abf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:49:14.870920 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:14.870801 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad40b959-18ef-44f8-a81e-d33e19794abf-istio-envoy" pod "ad40b959-18ef-44f8-a81e-d33e19794abf" (UID: "ad40b959-18ef-44f8-a81e-d33e19794abf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:49:14.871200 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:14.871172 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ad40b959-18ef-44f8-a81e-d33e19794abf-istio-podinfo" pod "ad40b959-18ef-44f8-a81e-d33e19794abf" (UID: "ad40b959-18ef-44f8-a81e-d33e19794abf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:49:14.871290 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:14.871213 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad40b959-18ef-44f8-a81e-d33e19794abf-kube-api-access-qbpvb" pod "ad40b959-18ef-44f8-a81e-d33e19794abf" (UID: "ad40b959-18ef-44f8-a81e-d33e19794abf"). InnerVolumeSpecName "kube-api-access-qbpvb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:49:14.896528 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:14.896499 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lrcmd" Nov 03 01:49:14.898959 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:14.898938 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-8p8gc" Nov 03 01:49:14.970093 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:14.970053 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a5060a0-9abf-4586-aef4-8d681a13b2aa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a5060a0-9abf-4586-aef4-8d681a13b2aa-workload-socket\") pod \"3a5060a0-9abf-4586-aef4-8d681a13b2aa\" (UID: \"3a5060a0-9abf-4586-aef4-8d681a13b2aa\") " Nov 03 01:49:14.970093 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:14.970101 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a5060a0-9abf-4586-aef4-8d681a13b2aa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a5060a0-9abf-4586-aef4-8d681a13b2aa-istiod-ca-cert\") pod \"3a5060a0-9abf-4586-aef4-8d681a13b2aa\" (UID: \"3a5060a0-9abf-4586-aef4-8d681a13b2aa\") " Nov 03 01:49:14.970398 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:14.970127 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/296123df-fd9f-4268-a61c-bb6ee52240be-istio-token\" (UniqueName: \"kubernetes.io/projected/296123df-fd9f-4268-a61c-bb6ee52240be-istio-token\") pod \"296123df-fd9f-4268-a61c-bb6ee52240be\" (UID: \"296123df-fd9f-4268-a61c-bb6ee52240be\") " Nov 03 01:49:14.970398 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:14.970145 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a5060a0-9abf-4586-aef4-8d681a13b2aa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a5060a0-9abf-4586-aef4-8d681a13b2aa-istio-data\") pod \"3a5060a0-9abf-4586-aef4-8d681a13b2aa\" (UID: \"3a5060a0-9abf-4586-aef4-8d681a13b2aa\") " Nov 03 01:49:14.970398 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:14.970231 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/296123df-fd9f-4268-a61c-bb6ee52240be-kube-api-access-wgb6h\" (UniqueName: \"kubernetes.io/projected/296123df-fd9f-4268-a61c-bb6ee52240be-kube-api-access-wgb6h\") pod \"296123df-fd9f-4268-a61c-bb6ee52240be\" (UID: \"296123df-fd9f-4268-a61c-bb6ee52240be\") " Nov 03 01:49:14.970398 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:14.970287 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/296123df-fd9f-4268-a61c-bb6ee52240be-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/296123df-fd9f-4268-a61c-bb6ee52240be-istio-podinfo\") pod \"296123df-fd9f-4268-a61c-bb6ee52240be\" (UID: \"296123df-fd9f-4268-a61c-bb6ee52240be\") " Nov 03 01:49:14.970398 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:14.970311 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a5060a0-9abf-4586-aef4-8d681a13b2aa-istio-token\" (UniqueName: \"kubernetes.io/projected/3a5060a0-9abf-4586-aef4-8d681a13b2aa-istio-token\") pod \"3a5060a0-9abf-4586-aef4-8d681a13b2aa\" (UID: \"3a5060a0-9abf-4586-aef4-8d681a13b2aa\") " Nov 03 01:49:14.970398 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:14.970350 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/296123df-fd9f-4268-a61c-bb6ee52240be-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/296123df-fd9f-4268-a61c-bb6ee52240be-istio-envoy\") pod \"296123df-fd9f-4268-a61c-bb6ee52240be\" (UID: \"296123df-fd9f-4268-a61c-bb6ee52240be\") " Nov 03 01:49:14.970765 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:14.970424 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a5060a0-9abf-4586-aef4-8d681a13b2aa-kube-api-access-wfcqq\" (UniqueName: \"kubernetes.io/projected/3a5060a0-9abf-4586-aef4-8d681a13b2aa-kube-api-access-wfcqq\") pod \"3a5060a0-9abf-4586-aef4-8d681a13b2aa\" (UID: \"3a5060a0-9abf-4586-aef4-8d681a13b2aa\") " Nov 03 01:49:14.970765 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:14.970511 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/296123df-fd9f-4268-a61c-bb6ee52240be-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/296123df-fd9f-4268-a61c-bb6ee52240be-istio-data\") pod \"296123df-fd9f-4268-a61c-bb6ee52240be\" (UID: \"296123df-fd9f-4268-a61c-bb6ee52240be\") " Nov 03 01:49:14.970765 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:14.970543 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/296123df-fd9f-4268-a61c-bb6ee52240be-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/296123df-fd9f-4268-a61c-bb6ee52240be-istiod-ca-cert\") pod \"296123df-fd9f-4268-a61c-bb6ee52240be\" (UID: \"296123df-fd9f-4268-a61c-bb6ee52240be\") " Nov 03 01:49:14.970765 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:14.970573 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a5060a0-9abf-4586-aef4-8d681a13b2aa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a5060a0-9abf-4586-aef4-8d681a13b2aa-istio-envoy\") pod \"3a5060a0-9abf-4586-aef4-8d681a13b2aa\" (UID: \"3a5060a0-9abf-4586-aef4-8d681a13b2aa\") " Nov 03 01:49:14.970765 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:14.970609 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a5060a0-9abf-4586-aef4-8d681a13b2aa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a5060a0-9abf-4586-aef4-8d681a13b2aa-credential-socket\") pod \"3a5060a0-9abf-4586-aef4-8d681a13b2aa\" (UID: \"3a5060a0-9abf-4586-aef4-8d681a13b2aa\") " Nov 03 01:49:14.970765 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:14.970668 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3a5060a0-9abf-4586-aef4-8d681a13b2aa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a5060a0-9abf-4586-aef4-8d681a13b2aa-istio-podinfo\") pod \"3a5060a0-9abf-4586-aef4-8d681a13b2aa\" (UID: \"3a5060a0-9abf-4586-aef4-8d681a13b2aa\") " Nov 03 01:49:14.970765 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:14.970700 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/296123df-fd9f-4268-a61c-bb6ee52240be-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/296123df-fd9f-4268-a61c-bb6ee52240be-credential-socket\") pod \"296123df-fd9f-4268-a61c-bb6ee52240be\" (UID: \"296123df-fd9f-4268-a61c-bb6ee52240be\") " Nov 03 01:49:14.970765 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:14.970740 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/296123df-fd9f-4268-a61c-bb6ee52240be-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/296123df-fd9f-4268-a61c-bb6ee52240be-workload-socket\") pod \"296123df-fd9f-4268-a61c-bb6ee52240be\" (UID: \"296123df-fd9f-4268-a61c-bb6ee52240be\") " Nov 03 01:49:14.971171 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:14.970418 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a5060a0-9abf-4586-aef4-8d681a13b2aa-workload-socket" pod "3a5060a0-9abf-4586-aef4-8d681a13b2aa" (UID: "3a5060a0-9abf-4586-aef4-8d681a13b2aa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:49:14.971171 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:14.970595 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a5060a0-9abf-4586-aef4-8d681a13b2aa-istio-data" pod "3a5060a0-9abf-4586-aef4-8d681a13b2aa" (UID: "3a5060a0-9abf-4586-aef4-8d681a13b2aa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:49:14.971171 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:14.970775 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/296123df-fd9f-4268-a61c-bb6ee52240be-istio-data" pod "296123df-fd9f-4268-a61c-bb6ee52240be" (UID: "296123df-fd9f-4268-a61c-bb6ee52240be"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:49:14.971171 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:14.970830 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a5060a0-9abf-4586-aef4-8d681a13b2aa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a5060a0-9abf-4586-aef4-8d681a13b2aa-workload-certs\") pod \"3a5060a0-9abf-4586-aef4-8d681a13b2aa\" (UID: \"3a5060a0-9abf-4586-aef4-8d681a13b2aa\") " Nov 03 01:49:14.971171 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:14.970873 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/296123df-fd9f-4268-a61c-bb6ee52240be-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/296123df-fd9f-4268-a61c-bb6ee52240be-workload-certs\") pod \"296123df-fd9f-4268-a61c-bb6ee52240be\" (UID: \"296123df-fd9f-4268-a61c-bb6ee52240be\") " Nov 03 01:49:14.971171 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:14.971000 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a5060a0-9abf-4586-aef4-8d681a13b2aa-credential-socket" pod "3a5060a0-9abf-4586-aef4-8d681a13b2aa" (UID: "3a5060a0-9abf-4586-aef4-8d681a13b2aa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:49:14.971171 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:14.971020 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a5060a0-9abf-4586-aef4-8d681a13b2aa-istiod-ca-cert" pod "3a5060a0-9abf-4586-aef4-8d681a13b2aa" (UID: "3a5060a0-9abf-4586-aef4-8d681a13b2aa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:49:14.971522 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:14.971344 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/296123df-fd9f-4268-a61c-bb6ee52240be-istiod-ca-cert" pod "296123df-fd9f-4268-a61c-bb6ee52240be" (UID: "296123df-fd9f-4268-a61c-bb6ee52240be"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:49:14.971522 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:14.971359 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/296123df-fd9f-4268-a61c-bb6ee52240be-credential-socket" pod "296123df-fd9f-4268-a61c-bb6ee52240be" (UID: "296123df-fd9f-4268-a61c-bb6ee52240be"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:49:14.971636 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:14.971529 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/296123df-fd9f-4268-a61c-bb6ee52240be-workload-socket" pod "296123df-fd9f-4268-a61c-bb6ee52240be" (UID: "296123df-fd9f-4268-a61c-bb6ee52240be"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:49:14.971636 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:14.971550 5437 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a5060a0-9abf-4586-aef4-8d681a13b2aa-workload-socket\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:49:14.971636 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:14.971586 5437 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a5060a0-9abf-4586-aef4-8d681a13b2aa-istiod-ca-cert\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:49:14.971636 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:14.971605 5437 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a5060a0-9abf-4586-aef4-8d681a13b2aa-istio-data\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:49:14.971636 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:14.971620 5437 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad40b959-18ef-44f8-a81e-d33e19794abf-workload-certs\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:49:14.971636 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:14.971628 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a5060a0-9abf-4586-aef4-8d681a13b2aa-workload-certs" pod "3a5060a0-9abf-4586-aef4-8d681a13b2aa" (UID: "3a5060a0-9abf-4586-aef4-8d681a13b2aa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:49:14.971636 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:14.971634 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbpvb\" (UniqueName: \"kubernetes.io/projected/ad40b959-18ef-44f8-a81e-d33e19794abf-kube-api-access-qbpvb\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:49:14.972011 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:14.971651 5437 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad40b959-18ef-44f8-a81e-d33e19794abf-istio-token\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:49:14.972011 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:14.971666 5437 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad40b959-18ef-44f8-a81e-d33e19794abf-istio-podinfo\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:49:14.972011 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:14.971680 5437 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/296123df-fd9f-4268-a61c-bb6ee52240be-istio-data\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:49:14.972011 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:14.971694 5437 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/296123df-fd9f-4268-a61c-bb6ee52240be-istiod-ca-cert\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:49:14.972011 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:14.971707 5437 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a5060a0-9abf-4586-aef4-8d681a13b2aa-credential-socket\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:49:14.972011 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:14.971721 5437 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/296123df-fd9f-4268-a61c-bb6ee52240be-credential-socket\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:49:14.972011 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:14.971735 5437 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad40b959-18ef-44f8-a81e-d33e19794abf-istiod-ca-cert\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:49:14.972011 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:14.971749 5437 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad40b959-18ef-44f8-a81e-d33e19794abf-istio-envoy\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:49:14.972011 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:14.971763 5437 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad40b959-18ef-44f8-a81e-d33e19794abf-credential-socket\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:49:14.972011 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:14.971775 5437 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad40b959-18ef-44f8-a81e-d33e19794abf-istio-data\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:49:14.972011 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:14.971733 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/296123df-fd9f-4268-a61c-bb6ee52240be-workload-certs" pod "296123df-fd9f-4268-a61c-bb6ee52240be" (UID: "296123df-fd9f-4268-a61c-bb6ee52240be"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:49:14.973427 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:14.973287 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/296123df-fd9f-4268-a61c-bb6ee52240be-istio-envoy" pod "296123df-fd9f-4268-a61c-bb6ee52240be" (UID: "296123df-fd9f-4268-a61c-bb6ee52240be"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:49:14.973559 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:14.973446 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/296123df-fd9f-4268-a61c-bb6ee52240be-kube-api-access-wgb6h" pod "296123df-fd9f-4268-a61c-bb6ee52240be" (UID: "296123df-fd9f-4268-a61c-bb6ee52240be"). InnerVolumeSpecName "kube-api-access-wgb6h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:49:14.973629 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:14.973568 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/296123df-fd9f-4268-a61c-bb6ee52240be-istio-token" pod "296123df-fd9f-4268-a61c-bb6ee52240be" (UID: "296123df-fd9f-4268-a61c-bb6ee52240be"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:49:14.974057 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:14.974028 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/296123df-fd9f-4268-a61c-bb6ee52240be-istio-podinfo" pod "296123df-fd9f-4268-a61c-bb6ee52240be" (UID: "296123df-fd9f-4268-a61c-bb6ee52240be"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:49:14.974144 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:14.974064 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a5060a0-9abf-4586-aef4-8d681a13b2aa-istio-token" pod "3a5060a0-9abf-4586-aef4-8d681a13b2aa" (UID: "3a5060a0-9abf-4586-aef4-8d681a13b2aa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:49:14.974336 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:14.974318 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a5060a0-9abf-4586-aef4-8d681a13b2aa-istio-envoy" pod "3a5060a0-9abf-4586-aef4-8d681a13b2aa" (UID: "3a5060a0-9abf-4586-aef4-8d681a13b2aa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:49:14.974540 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:14.974523 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3a5060a0-9abf-4586-aef4-8d681a13b2aa-istio-podinfo" pod "3a5060a0-9abf-4586-aef4-8d681a13b2aa" (UID: "3a5060a0-9abf-4586-aef4-8d681a13b2aa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:49:14.974725 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:14.974710 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a5060a0-9abf-4586-aef4-8d681a13b2aa-kube-api-access-wfcqq" pod "3a5060a0-9abf-4586-aef4-8d681a13b2aa" (UID: "3a5060a0-9abf-4586-aef4-8d681a13b2aa"). InnerVolumeSpecName "kube-api-access-wfcqq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:49:15.073328 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:15.073228 5437 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/296123df-fd9f-4268-a61c-bb6ee52240be-istio-podinfo\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:49:15.073328 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:15.073274 5437 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a5060a0-9abf-4586-aef4-8d681a13b2aa-istio-token\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:49:15.073328 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:15.073284 5437 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/296123df-fd9f-4268-a61c-bb6ee52240be-istio-envoy\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:49:15.073328 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:15.073295 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfcqq\" (UniqueName: \"kubernetes.io/projected/3a5060a0-9abf-4586-aef4-8d681a13b2aa-kube-api-access-wfcqq\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:49:15.073328 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:15.073305 5437 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a5060a0-9abf-4586-aef4-8d681a13b2aa-istio-envoy\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:49:15.073328 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:15.073316 5437 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a5060a0-9abf-4586-aef4-8d681a13b2aa-istio-podinfo\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:49:15.073328 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:15.073325 5437 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/296123df-fd9f-4268-a61c-bb6ee52240be-workload-socket\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:49:15.073328 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:15.073334 5437 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a5060a0-9abf-4586-aef4-8d681a13b2aa-workload-certs\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:49:15.073328 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:15.073344 5437 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/296123df-fd9f-4268-a61c-bb6ee52240be-workload-certs\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:49:15.073958 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:15.073352 5437 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/296123df-fd9f-4268-a61c-bb6ee52240be-istio-token\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:49:15.073958 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:15.073378 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgb6h\" (UniqueName: \"kubernetes.io/projected/296123df-fd9f-4268-a61c-bb6ee52240be-kube-api-access-wgb6h\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:49:15.635962 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:15.635927 5437 generic.go:356] "Generic (PLEG): container finished" podID="3a5060a0-9abf-4586-aef4-8d681a13b2aa" containerID="48cf9404fc106d59f6c5cf3c6cac07d0da8ab76c409859f9b5f7f6319666d8dd" exitCode=0 Nov 03 01:49:15.636166 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:15.635974 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-8p8gc" Nov 03 01:49:15.636166 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:15.636018 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-8p8gc" event={"ID":"3a5060a0-9abf-4586-aef4-8d681a13b2aa","Type":"ContainerDied","Data":"48cf9404fc106d59f6c5cf3c6cac07d0da8ab76c409859f9b5f7f6319666d8dd"} Nov 03 01:49:15.636166 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:15.636075 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-8p8gc" event={"ID":"3a5060a0-9abf-4586-aef4-8d681a13b2aa","Type":"ContainerDied","Data":"d45ab1d114c2efd510f9c90364d2be7b2744d017b69a96b40190ce408908528c"} Nov 03 01:49:15.636166 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:15.636103 5437 scope.go:122] "RemoveContainer" containerID="48cf9404fc106d59f6c5cf3c6cac07d0da8ab76c409859f9b5f7f6319666d8dd" Nov 03 01:49:15.637578 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:15.637550 5437 generic.go:356] "Generic (PLEG): container finished" podID="296123df-fd9f-4268-a61c-bb6ee52240be" containerID="abb436119f52563e452c26ff2e2478d0a16d35cb488a27a84ed15e12f6ddb21c" exitCode=0 Nov 03 01:49:15.637694 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:15.637590 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lrcmd" Nov 03 01:49:15.637694 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:15.637634 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lrcmd" event={"ID":"296123df-fd9f-4268-a61c-bb6ee52240be","Type":"ContainerDied","Data":"abb436119f52563e452c26ff2e2478d0a16d35cb488a27a84ed15e12f6ddb21c"} Nov 03 01:49:15.637694 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:15.637679 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lrcmd" event={"ID":"296123df-fd9f-4268-a61c-bb6ee52240be","Type":"ContainerDied","Data":"4eefaed904e675fcfe3645219daa6577a31560fdf12ac5d32af7a7d896caf226"} Nov 03 01:49:15.639208 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:15.639183 5437 generic.go:356] "Generic (PLEG): container finished" podID="ad40b959-18ef-44f8-a81e-d33e19794abf" containerID="56f673e8057177a93349f206f1d2e80bf649739f9f562e6e72f29bda0ad5ce85" exitCode=0 Nov 03 01:49:15.639328 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:15.639224 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lhxlr" Nov 03 01:49:15.639328 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:15.639231 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lhxlr" event={"ID":"ad40b959-18ef-44f8-a81e-d33e19794abf","Type":"ContainerDied","Data":"56f673e8057177a93349f206f1d2e80bf649739f9f562e6e72f29bda0ad5ce85"} Nov 03 01:49:15.639328 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:15.639274 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lhxlr" event={"ID":"ad40b959-18ef-44f8-a81e-d33e19794abf","Type":"ContainerDied","Data":"50773111940d6c5bb867e9146357f7cb8f777f5531e98bf13b4191325ed9b7f1"} Nov 03 01:49:15.646976 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:15.646947 5437 scope.go:122] "RemoveContainer" containerID="48cf9404fc106d59f6c5cf3c6cac07d0da8ab76c409859f9b5f7f6319666d8dd" Nov 03 01:49:15.647333 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:49:15.647310 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48cf9404fc106d59f6c5cf3c6cac07d0da8ab76c409859f9b5f7f6319666d8dd\": container with ID starting with 48cf9404fc106d59f6c5cf3c6cac07d0da8ab76c409859f9b5f7f6319666d8dd not found: ID does not exist" containerID="48cf9404fc106d59f6c5cf3c6cac07d0da8ab76c409859f9b5f7f6319666d8dd" Nov 03 01:49:15.647495 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:15.647345 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48cf9404fc106d59f6c5cf3c6cac07d0da8ab76c409859f9b5f7f6319666d8dd"} err="failed to get container status \"48cf9404fc106d59f6c5cf3c6cac07d0da8ab76c409859f9b5f7f6319666d8dd\": rpc error: code = NotFound desc = could not find container \"48cf9404fc106d59f6c5cf3c6cac07d0da8ab76c409859f9b5f7f6319666d8dd\": container with ID starting with 48cf9404fc106d59f6c5cf3c6cac07d0da8ab76c409859f9b5f7f6319666d8dd not found: ID does not exist" Nov 03 01:49:15.647495 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:15.647488 5437 scope.go:122] "RemoveContainer" containerID="abb436119f52563e452c26ff2e2478d0a16d35cb488a27a84ed15e12f6ddb21c" Nov 03 01:49:15.657330 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:15.657225 5437 scope.go:122] "RemoveContainer" containerID="abb436119f52563e452c26ff2e2478d0a16d35cb488a27a84ed15e12f6ddb21c" Nov 03 01:49:15.657577 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:49:15.657557 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abb436119f52563e452c26ff2e2478d0a16d35cb488a27a84ed15e12f6ddb21c\": container with ID starting with abb436119f52563e452c26ff2e2478d0a16d35cb488a27a84ed15e12f6ddb21c not found: ID does not exist" containerID="abb436119f52563e452c26ff2e2478d0a16d35cb488a27a84ed15e12f6ddb21c" Nov 03 01:49:15.657637 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:15.657589 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abb436119f52563e452c26ff2e2478d0a16d35cb488a27a84ed15e12f6ddb21c"} err="failed to get container status \"abb436119f52563e452c26ff2e2478d0a16d35cb488a27a84ed15e12f6ddb21c\": rpc error: code = NotFound desc = could not find container \"abb436119f52563e452c26ff2e2478d0a16d35cb488a27a84ed15e12f6ddb21c\": container with ID starting with abb436119f52563e452c26ff2e2478d0a16d35cb488a27a84ed15e12f6ddb21c not found: ID does not exist" Nov 03 01:49:15.657637 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:15.657609 5437 scope.go:122] "RemoveContainer" containerID="56f673e8057177a93349f206f1d2e80bf649739f9f562e6e72f29bda0ad5ce85" Nov 03 01:49:15.666854 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:15.666829 5437 scope.go:122] "RemoveContainer" containerID="56f673e8057177a93349f206f1d2e80bf649739f9f562e6e72f29bda0ad5ce85" Nov 03 01:49:15.667199 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:49:15.667179 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56f673e8057177a93349f206f1d2e80bf649739f9f562e6e72f29bda0ad5ce85\": container with ID starting with 56f673e8057177a93349f206f1d2e80bf649739f9f562e6e72f29bda0ad5ce85 not found: ID does not exist" containerID="56f673e8057177a93349f206f1d2e80bf649739f9f562e6e72f29bda0ad5ce85" Nov 03 01:49:15.667260 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:15.667208 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56f673e8057177a93349f206f1d2e80bf649739f9f562e6e72f29bda0ad5ce85"} err="failed to get container status \"56f673e8057177a93349f206f1d2e80bf649739f9f562e6e72f29bda0ad5ce85\": rpc error: code = NotFound desc = could not find container \"56f673e8057177a93349f206f1d2e80bf649739f9f562e6e72f29bda0ad5ce85\": container with ID starting with 56f673e8057177a93349f206f1d2e80bf649739f9f562e6e72f29bda0ad5ce85 not found: ID does not exist" Nov 03 01:49:15.807866 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:15.807828 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lrcmd"] Nov 03 01:49:15.855236 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:15.855198 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:49:15.859145 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:15.859115 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lrcmd"] Nov 03 01:49:15.928302 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:15.928258 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lhxlr"] Nov 03 01:49:15.937868 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:15.937826 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-lhxlr"] Nov 03 01:49:16.051910 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:16.051868 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-8p8gc"] Nov 03 01:49:16.055805 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:16.055773 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:49:16.056534 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:16.056514 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-8p8gc"] Nov 03 01:49:16.095224 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:16.095141 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="296123df-fd9f-4268-a61c-bb6ee52240be" path="/var/lib/kubelet/pods/296123df-fd9f-4268-a61c-bb6ee52240be/volumes" Nov 03 01:49:16.095960 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:16.095938 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a5060a0-9abf-4586-aef4-8d681a13b2aa" path="/var/lib/kubelet/pods/3a5060a0-9abf-4586-aef4-8d681a13b2aa/volumes" Nov 03 01:49:16.096640 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:16.096622 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad40b959-18ef-44f8-a81e-d33e19794abf" path="/var/lib/kubelet/pods/ad40b959-18ef-44f8-a81e-d33e19794abf/volumes" Nov 03 01:49:16.263217 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:16.263177 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:49:19.070970 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:19.070903 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:49:19.277396 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:19.277325 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:49:21.228715 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:21.228669 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:49:25.101883 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:25.101834 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:49:25.647537 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:25.647496 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:49:26.027127 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:26.027027 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:49:26.721135 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:26.721099 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:49:26.923976 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:26.923908 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:49:27.126506 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:27.126471 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:49:27.324505 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:27.324470 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:49:27.532903 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:27.532810 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:49:31.897703 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:31.897662 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:49:32.109390 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:32.109337 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:49:32.322504 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:32.322471 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:49:32.655171 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:32.655071 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:49:35.107871 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:35.107828 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:49:35.313972 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:35.313930 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:49:37.053684 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:37.053645 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:49:37.375755 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:37.375654 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:49:37.985684 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:37.985645 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:49:38.191845 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:38.191803 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:49:38.397240 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:38.397201 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:49:39.138246 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:39.138204 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:49:39.645870 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:39.645836 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:49:40.633728 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:40.633691 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:49:41.256156 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:41.256120 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:49:41.468776 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:41.468742 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:49:43.211908 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:43.211864 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:49:44.036567 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:49:44.036524 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 01:49:44.622352 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:44.622303 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:49:44.824875 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:44.824833 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:49:45.327187 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:45.327150 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:49:45.586987 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:45.586887 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:49:46.223529 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:46.223486 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:49:48.403151 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:48.403107 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:49:48.604190 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:48.604154 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:49:48.811259 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:48.811220 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:49:51.029341 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:51.029301 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:49:53.687576 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:53.687533 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:49:57.382614 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:57.382559 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:49:57.898186 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:49:57.898146 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:50:00.578633 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:00.578597 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:50:05.226327 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:05.226290 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:50:05.481610 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:05.481506 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:50:10.306935 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:10.306889 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:50:13.719908 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:13.719818 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:50:14.037590 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:50:14.037491 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 01:50:14.195898 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:14.195842 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:50:16.866107 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:16.866022 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:50:20.895345 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:20.895299 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:50:21.519688 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:21.519647 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:50:21.869971 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:21.869930 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:50:22.123708 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:22.123613 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:50:25.621743 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:25.621705 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:50:25.883213 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:25.883112 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:50:26.467963 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:26.467913 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:50:26.673813 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:26.673767 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:50:26.875598 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:26.875562 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:50:29.570013 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:29.569979 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:50:29.776832 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:29.776794 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:50:29.905527 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:29.905435 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-71746a-6f445fbc6c-sbmzg"] Nov 03 01:50:29.906030 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:29.906012 5437 cpu_manager.go:414] "containerMap: removing container" podUID="6faa241e-9510-45af-9e59-e3aa35524217" containerName="payload-processing" Nov 03 01:50:29.906030 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:29.906029 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6faa241e-9510-45af-9e59-e3aa35524217" containerName="payload-processing" Nov 03 01:50:29.906189 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:29.906046 5437 cpu_manager.go:414] "containerMap: removing container" podUID="3a5060a0-9abf-4586-aef4-8d681a13b2aa" containerName="istio-proxy" Nov 03 01:50:29.906189 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:29.906051 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a5060a0-9abf-4586-aef4-8d681a13b2aa" containerName="istio-proxy" Nov 03 01:50:29.906189 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:29.906067 5437 cpu_manager.go:414] "containerMap: removing container" podUID="ad40b959-18ef-44f8-a81e-d33e19794abf" containerName="istio-proxy" Nov 03 01:50:29.906189 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:29.906072 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad40b959-18ef-44f8-a81e-d33e19794abf" containerName="istio-proxy" Nov 03 01:50:29.906189 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:29.906083 5437 cpu_manager.go:414] "containerMap: removing container" podUID="5b441095-b0c0-4bbe-89af-5cf30727a4a7" containerName="istio-proxy" Nov 03 01:50:29.906189 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:29.906088 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b441095-b0c0-4bbe-89af-5cf30727a4a7" containerName="istio-proxy" Nov 03 01:50:29.906189 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:29.906097 5437 cpu_manager.go:414] "containerMap: removing container" podUID="87a89600-532d-4c7a-99f4-a9d41a58bfc5" containerName="maas-api" Nov 03 01:50:29.906189 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:29.906101 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87a89600-532d-4c7a-99f4-a9d41a58bfc5" containerName="maas-api" Nov 03 01:50:29.906189 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:29.906112 5437 cpu_manager.go:414] "containerMap: removing container" podUID="ea981b45-743a-4e06-83c9-519fbda3d7d8" containerName="istio-proxy" Nov 03 01:50:29.906189 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:29.906117 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea981b45-743a-4e06-83c9-519fbda3d7d8" containerName="istio-proxy" Nov 03 01:50:29.906189 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:29.906131 5437 cpu_manager.go:414] "containerMap: removing container" podUID="a3547027-5292-4db3-8d5e-fa7e0f792d8f" containerName="payload-pre-processing" Nov 03 01:50:29.906189 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:29.906136 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3547027-5292-4db3-8d5e-fa7e0f792d8f" containerName="payload-pre-processing" Nov 03 01:50:29.906189 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:29.906147 5437 cpu_manager.go:414] "containerMap: removing container" podUID="b8ce6d38-37d9-45ef-a6c0-4f7ad8733998" containerName="maas-api" Nov 03 01:50:29.906189 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:29.906152 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8ce6d38-37d9-45ef-a6c0-4f7ad8733998" containerName="maas-api" Nov 03 01:50:29.906189 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:29.906161 5437 cpu_manager.go:414] "containerMap: removing container" podUID="296123df-fd9f-4268-a61c-bb6ee52240be" containerName="istio-proxy" Nov 03 01:50:29.906189 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:29.906165 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="296123df-fd9f-4268-a61c-bb6ee52240be" containerName="istio-proxy" Nov 03 01:50:29.906825 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:29.906240 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a5060a0-9abf-4586-aef4-8d681a13b2aa" containerName="istio-proxy" Nov 03 01:50:29.906825 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:29.906252 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad40b959-18ef-44f8-a81e-d33e19794abf" containerName="istio-proxy" Nov 03 01:50:29.906825 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:29.906258 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b441095-b0c0-4bbe-89af-5cf30727a4a7" containerName="istio-proxy" Nov 03 01:50:29.906825 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:29.906264 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="87a89600-532d-4c7a-99f4-a9d41a58bfc5" containerName="maas-api" Nov 03 01:50:29.906825 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:29.906271 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea981b45-743a-4e06-83c9-519fbda3d7d8" containerName="istio-proxy" Nov 03 01:50:29.906825 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:29.906279 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3547027-5292-4db3-8d5e-fa7e0f792d8f" containerName="payload-pre-processing" Nov 03 01:50:29.906825 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:29.906284 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8ce6d38-37d9-45ef-a6c0-4f7ad8733998" containerName="maas-api" Nov 03 01:50:29.906825 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:29.906291 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="296123df-fd9f-4268-a61c-bb6ee52240be" containerName="istio-proxy" Nov 03 01:50:29.906825 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:29.906297 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="6faa241e-9510-45af-9e59-e3aa35524217" containerName="payload-processing" Nov 03 01:50:29.910027 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:29.910000 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-71746a-6f445fbc6c-sbmzg" Nov 03 01:50:30.016255 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:30.016216 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-71746a\"" Nov 03 01:50:30.016498 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:30.016309 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-71746a-dockercfg-79hcx\"" Nov 03 01:50:30.020646 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:30.020606 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-71746a-6f445fbc6c-sbmzg"] Nov 03 01:50:30.077494 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:30.077454 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c0b0e96-c609-4127-ab55-c0c8f11e4e01-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-71746a-6f445fbc6c-sbmzg\" (UID: \"4c0b0e96-c609-4127-ab55-c0c8f11e4e01\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-71746a-6f445fbc6c-sbmzg" Nov 03 01:50:30.082476 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:30.082441 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76znl\" (UniqueName: \"kubernetes.io/projected/4c0b0e96-c609-4127-ab55-c0c8f11e4e01-kube-api-access-76znl\") pod \"payload-pre-processing-e2e-ipp-b-71746a-6f445fbc6c-sbmzg\" (UID: \"4c0b0e96-c609-4127-ab55-c0c8f11e4e01\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-71746a-6f445fbc6c-sbmzg" Nov 03 01:50:30.104318 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:30.104292 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-71746a-86854c59ff-x7lbf"] Nov 03 01:50:30.108704 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:30.108677 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-71746a-86854c59ff-x7lbf" Nov 03 01:50:30.140480 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:30.140446 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-71746a-86854c59ff-x7lbf"] Nov 03 01:50:30.184214 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:30.184122 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c0b0e96-c609-4127-ab55-c0c8f11e4e01-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-71746a-6f445fbc6c-sbmzg\" (UID: \"4c0b0e96-c609-4127-ab55-c0c8f11e4e01\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-71746a-6f445fbc6c-sbmzg" Nov 03 01:50:30.184441 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:30.184268 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76znl\" (UniqueName: \"kubernetes.io/projected/4c0b0e96-c609-4127-ab55-c0c8f11e4e01-kube-api-access-76znl\") pod \"payload-pre-processing-e2e-ipp-b-71746a-6f445fbc6c-sbmzg\" (UID: \"4c0b0e96-c609-4127-ab55-c0c8f11e4e01\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-71746a-6f445fbc6c-sbmzg" Nov 03 01:50:30.184574 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:30.184553 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6f96330-e093-4c72-9501-83baed0edee8-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-71746a-86854c59ff-x7lbf\" (UID: \"d6f96330-e093-4c72-9501-83baed0edee8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-71746a-86854c59ff-x7lbf" Nov 03 01:50:30.184939 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:30.184902 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmqgq\" (UniqueName: \"kubernetes.io/projected/d6f96330-e093-4c72-9501-83baed0edee8-kube-api-access-dmqgq\") pod \"payload-processing-e2e-ipp-b-71746a-86854c59ff-x7lbf\" (UID: \"d6f96330-e093-4c72-9501-83baed0edee8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-71746a-86854c59ff-x7lbf" Nov 03 01:50:30.185066 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:30.184999 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c0b0e96-c609-4127-ab55-c0c8f11e4e01-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-71746a-6f445fbc6c-sbmzg\" (UID: \"4c0b0e96-c609-4127-ab55-c0c8f11e4e01\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-71746a-6f445fbc6c-sbmzg" Nov 03 01:50:30.247473 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:30.247433 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76znl\" (UniqueName: \"kubernetes.io/projected/4c0b0e96-c609-4127-ab55-c0c8f11e4e01-kube-api-access-76znl\") pod \"payload-pre-processing-e2e-ipp-b-71746a-6f445fbc6c-sbmzg\" (UID: \"4c0b0e96-c609-4127-ab55-c0c8f11e4e01\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-71746a-6f445fbc6c-sbmzg" Nov 03 01:50:30.286619 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:30.286580 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmqgq\" (UniqueName: \"kubernetes.io/projected/d6f96330-e093-4c72-9501-83baed0edee8-kube-api-access-dmqgq\") pod \"payload-processing-e2e-ipp-b-71746a-86854c59ff-x7lbf\" (UID: \"d6f96330-e093-4c72-9501-83baed0edee8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-71746a-86854c59ff-x7lbf" Nov 03 01:50:30.286840 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:30.286731 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6f96330-e093-4c72-9501-83baed0edee8-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-71746a-86854c59ff-x7lbf\" (UID: \"d6f96330-e093-4c72-9501-83baed0edee8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-71746a-86854c59ff-x7lbf" Nov 03 01:50:30.287664 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:30.287638 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6f96330-e093-4c72-9501-83baed0edee8-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-71746a-86854c59ff-x7lbf\" (UID: \"d6f96330-e093-4c72-9501-83baed0edee8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-71746a-86854c59ff-x7lbf" Nov 03 01:50:30.350998 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:30.350958 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmqgq\" (UniqueName: \"kubernetes.io/projected/d6f96330-e093-4c72-9501-83baed0edee8-kube-api-access-dmqgq\") pod \"payload-processing-e2e-ipp-b-71746a-86854c59ff-x7lbf\" (UID: \"d6f96330-e093-4c72-9501-83baed0edee8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-71746a-86854c59ff-x7lbf" Nov 03 01:50:30.424616 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:30.424582 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-71746a-86854c59ff-x7lbf" Nov 03 01:50:30.523694 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:30.523648 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-71746a-6f445fbc6c-sbmzg" Nov 03 01:50:30.811121 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:50:30.811082 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6f96330_e093_4c72_9501_83baed0edee8.slice/crio-91288973d584aa2be9a49b6b15a12c3b6b41fa5e372cc3da9ddafe762b3e4305 WatchSource:0}: Error finding container 91288973d584aa2be9a49b6b15a12c3b6b41fa5e372cc3da9ddafe762b3e4305: Status 404 returned error can't find the container with id 91288973d584aa2be9a49b6b15a12c3b6b41fa5e372cc3da9ddafe762b3e4305 Nov 03 01:50:30.814107 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:30.814075 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-71746a-86854c59ff-x7lbf"] Nov 03 01:50:30.915444 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:50:30.915406 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c0b0e96_c609_4127_ab55_c0c8f11e4e01.slice/crio-56a3b4cd59852f73d5eaf30a4456ef84636d1daa403dd60a03bd245a7276361f WatchSource:0}: Error finding container 56a3b4cd59852f73d5eaf30a4456ef84636d1daa403dd60a03bd245a7276361f: Status 404 returned error can't find the container with id 56a3b4cd59852f73d5eaf30a4456ef84636d1daa403dd60a03bd245a7276361f Nov 03 01:50:30.918084 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:30.918057 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-71746a-6f445fbc6c-sbmzg"] Nov 03 01:50:31.186194 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:31.186087 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-71746a-6f445fbc6c-sbmzg" event={"ID":"4c0b0e96-c609-4127-ab55-c0c8f11e4e01","Type":"ContainerStarted","Data":"9381d0f3c8c30c6f18de476dc08c44c75ad83e976608f606cdc413240e22da4d"} Nov 03 01:50:31.186194 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:31.186153 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-71746a-6f445fbc6c-sbmzg" event={"ID":"4c0b0e96-c609-4127-ab55-c0c8f11e4e01","Type":"ContainerStarted","Data":"56a3b4cd59852f73d5eaf30a4456ef84636d1daa403dd60a03bd245a7276361f"} Nov 03 01:50:31.186194 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:31.186187 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-71746a-6f445fbc6c-sbmzg" Nov 03 01:50:31.187853 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:31.187827 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-71746a-86854c59ff-x7lbf" event={"ID":"d6f96330-e093-4c72-9501-83baed0edee8","Type":"ContainerStarted","Data":"a4c83a294afd5791edde30dd8ab6bd43e0302342582d6c19fea6549d3173c62d"} Nov 03 01:50:31.187931 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:31.187877 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-71746a-86854c59ff-x7lbf" event={"ID":"d6f96330-e093-4c72-9501-83baed0edee8","Type":"ContainerStarted","Data":"91288973d584aa2be9a49b6b15a12c3b6b41fa5e372cc3da9ddafe762b3e4305"} Nov 03 01:50:31.187931 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:31.187898 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-71746a-86854c59ff-x7lbf" Nov 03 01:50:31.304942 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:31.304884 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-71746a-6f445fbc6c-sbmzg" podStartSLOduration=2.304869998 podStartE2EDuration="2.304869998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:50:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:50:31.304215748 +0000 UTC m=+1913.333698044" watchObservedRunningTime="2026-11-03 01:50:31.304869998 +0000 UTC m=+1913.334352292" Nov 03 01:50:31.424201 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:31.424143 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-71746a-86854c59ff-x7lbf" podStartSLOduration=1.424077428 podStartE2EDuration="1.424077428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:50:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:50:31.422992637 +0000 UTC m=+1913.452474933" watchObservedRunningTime="2026-11-03 01:50:31.424077428 +0000 UTC m=+1913.453559724" Nov 03 01:50:32.028868 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:32.028829 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:50:36.778344 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:36.778288 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:50:41.582323 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:41.582289 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:50:42.196760 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:42.196730 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-71746a-86854c59ff-x7lbf" Nov 03 01:50:42.196950 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:42.196812 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-71746a-6f445fbc6c-sbmzg" Nov 03 01:50:43.073747 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:43.073703 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:50:43.276819 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:43.276779 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:50:43.857847 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:43.857800 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-jxv68"] Nov 03 01:50:43.858198 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:43.858152 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-jxv68" podUID="5f118c89-c5d9-4939-b609-d3c173b27ae5" containerName="istio-proxy" containerID="cri-o://e37358d2a8a5f7ced043fd9922e3dc4634021ea0e09c7532cd0dd910f4054f0e" gracePeriod=30 Nov 03 01:50:44.036409 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:50:44.036339 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 01:50:44.399229 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:44.399187 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:50:45.329539 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:45.329493 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:50:45.767853 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:45.767763 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:50:46.083633 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:46.083597 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:50:46.291047 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:46.291008 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:50:46.546085 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:46.545984 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:50:46.749600 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:46.749565 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:50:46.953075 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:46.953037 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:50:47.238611 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:47.238526 5437 patch_prober.go:29] interesting pod/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-jxv68 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 01:50:47.238611 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:47.238593 5437 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-jxv68" podUID="5f118c89-c5d9-4939-b609-d3c173b27ae5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 01:50:49.120083 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:49.119999 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-jxv68" Nov 03 01:50:49.141965 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:49.141925 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f118c89-c5d9-4939-b609-d3c173b27ae5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f118c89-c5d9-4939-b609-d3c173b27ae5-istio-data\") pod \"5f118c89-c5d9-4939-b609-d3c173b27ae5\" (UID: \"5f118c89-c5d9-4939-b609-d3c173b27ae5\") " Nov 03 01:50:49.142165 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:49.142058 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f118c89-c5d9-4939-b609-d3c173b27ae5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f118c89-c5d9-4939-b609-d3c173b27ae5-credential-socket\") pod \"5f118c89-c5d9-4939-b609-d3c173b27ae5\" (UID: \"5f118c89-c5d9-4939-b609-d3c173b27ae5\") " Nov 03 01:50:49.142235 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:49.142162 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f118c89-c5d9-4939-b609-d3c173b27ae5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f118c89-c5d9-4939-b609-d3c173b27ae5-istio-envoy\") pod \"5f118c89-c5d9-4939-b609-d3c173b27ae5\" (UID: \"5f118c89-c5d9-4939-b609-d3c173b27ae5\") " Nov 03 01:50:49.142235 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:49.142224 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f118c89-c5d9-4939-b609-d3c173b27ae5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f118c89-c5d9-4939-b609-d3c173b27ae5-istiod-ca-cert\") pod \"5f118c89-c5d9-4939-b609-d3c173b27ae5\" (UID: \"5f118c89-c5d9-4939-b609-d3c173b27ae5\") " Nov 03 01:50:49.142330 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:49.142239 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f118c89-c5d9-4939-b609-d3c173b27ae5-istio-data" pod "5f118c89-c5d9-4939-b609-d3c173b27ae5" (UID: "5f118c89-c5d9-4939-b609-d3c173b27ae5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:50:49.142330 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:49.142267 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f118c89-c5d9-4939-b609-d3c173b27ae5-credential-socket" pod "5f118c89-c5d9-4939-b609-d3c173b27ae5" (UID: "5f118c89-c5d9-4939-b609-d3c173b27ae5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:50:49.142330 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:49.142281 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5f118c89-c5d9-4939-b609-d3c173b27ae5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f118c89-c5d9-4939-b609-d3c173b27ae5-istio-podinfo\") pod \"5f118c89-c5d9-4939-b609-d3c173b27ae5\" (UID: \"5f118c89-c5d9-4939-b609-d3c173b27ae5\") " Nov 03 01:50:49.142515 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:49.142339 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f118c89-c5d9-4939-b609-d3c173b27ae5-istio-token\" (UniqueName: \"kubernetes.io/projected/5f118c89-c5d9-4939-b609-d3c173b27ae5-istio-token\") pod \"5f118c89-c5d9-4939-b609-d3c173b27ae5\" (UID: \"5f118c89-c5d9-4939-b609-d3c173b27ae5\") " Nov 03 01:50:49.142515 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:49.142399 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f118c89-c5d9-4939-b609-d3c173b27ae5-kube-api-access-h2dsw\" (UniqueName: \"kubernetes.io/projected/5f118c89-c5d9-4939-b609-d3c173b27ae5-kube-api-access-h2dsw\") pod \"5f118c89-c5d9-4939-b609-d3c173b27ae5\" (UID: \"5f118c89-c5d9-4939-b609-d3c173b27ae5\") " Nov 03 01:50:49.142515 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:49.142451 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f118c89-c5d9-4939-b609-d3c173b27ae5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f118c89-c5d9-4939-b609-d3c173b27ae5-workload-socket\") pod \"5f118c89-c5d9-4939-b609-d3c173b27ae5\" (UID: \"5f118c89-c5d9-4939-b609-d3c173b27ae5\") " Nov 03 01:50:49.142661 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:49.142526 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f118c89-c5d9-4939-b609-d3c173b27ae5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f118c89-c5d9-4939-b609-d3c173b27ae5-workload-certs\") pod \"5f118c89-c5d9-4939-b609-d3c173b27ae5\" (UID: \"5f118c89-c5d9-4939-b609-d3c173b27ae5\") " Nov 03 01:50:49.142661 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:49.142618 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f118c89-c5d9-4939-b609-d3c173b27ae5-istiod-ca-cert" pod "5f118c89-c5d9-4939-b609-d3c173b27ae5" (UID: "5f118c89-c5d9-4939-b609-d3c173b27ae5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:50:49.142961 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:49.142883 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f118c89-c5d9-4939-b609-d3c173b27ae5-workload-socket" pod "5f118c89-c5d9-4939-b609-d3c173b27ae5" (UID: "5f118c89-c5d9-4939-b609-d3c173b27ae5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:50:49.143105 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:49.143010 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f118c89-c5d9-4939-b609-d3c173b27ae5-workload-certs" pod "5f118c89-c5d9-4939-b609-d3c173b27ae5" (UID: "5f118c89-c5d9-4939-b609-d3c173b27ae5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:50:49.143480 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:49.143451 5437 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f118c89-c5d9-4939-b609-d3c173b27ae5-istiod-ca-cert\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:50:49.143600 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:49.143483 5437 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f118c89-c5d9-4939-b609-d3c173b27ae5-workload-socket\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:50:49.143600 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:49.143498 5437 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f118c89-c5d9-4939-b609-d3c173b27ae5-workload-certs\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:50:49.143600 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:49.143514 5437 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f118c89-c5d9-4939-b609-d3c173b27ae5-istio-data\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:50:49.143600 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:49.143530 5437 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f118c89-c5d9-4939-b609-d3c173b27ae5-credential-socket\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:50:49.144840 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:49.144816 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f118c89-c5d9-4939-b609-d3c173b27ae5-istio-envoy" pod "5f118c89-c5d9-4939-b609-d3c173b27ae5" (UID: "5f118c89-c5d9-4939-b609-d3c173b27ae5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:50:49.144949 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:49.144872 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f118c89-c5d9-4939-b609-d3c173b27ae5-kube-api-access-h2dsw" pod "5f118c89-c5d9-4939-b609-d3c173b27ae5" (UID: "5f118c89-c5d9-4939-b609-d3c173b27ae5"). InnerVolumeSpecName "kube-api-access-h2dsw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:50:49.144999 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:49.144970 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f118c89-c5d9-4939-b609-d3c173b27ae5-istio-token" pod "5f118c89-c5d9-4939-b609-d3c173b27ae5" (UID: "5f118c89-c5d9-4939-b609-d3c173b27ae5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:50:49.144999 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:49.144970 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5f118c89-c5d9-4939-b609-d3c173b27ae5-istio-podinfo" pod "5f118c89-c5d9-4939-b609-d3c173b27ae5" (UID: "5f118c89-c5d9-4939-b609-d3c173b27ae5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:50:49.245106 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:49.245062 5437 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f118c89-c5d9-4939-b609-d3c173b27ae5-istio-podinfo\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:50:49.245106 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:49.245107 5437 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f118c89-c5d9-4939-b609-d3c173b27ae5-istio-token\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:50:49.245441 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:49.245124 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2dsw\" (UniqueName: \"kubernetes.io/projected/5f118c89-c5d9-4939-b609-d3c173b27ae5-kube-api-access-h2dsw\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:50:49.245441 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:49.245135 5437 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f118c89-c5d9-4939-b609-d3c173b27ae5-istio-envoy\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:50:49.302714 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:49.302676 5437 generic.go:356] "Generic (PLEG): container finished" podID="5f118c89-c5d9-4939-b609-d3c173b27ae5" containerID="e37358d2a8a5f7ced043fd9922e3dc4634021ea0e09c7532cd0dd910f4054f0e" exitCode=0 Nov 03 01:50:49.302879 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:49.302729 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-jxv68" Nov 03 01:50:49.302879 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:49.302749 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-jxv68" event={"ID":"5f118c89-c5d9-4939-b609-d3c173b27ae5","Type":"ContainerDied","Data":"e37358d2a8a5f7ced043fd9922e3dc4634021ea0e09c7532cd0dd910f4054f0e"} Nov 03 01:50:49.302879 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:49.302808 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-jxv68" event={"ID":"5f118c89-c5d9-4939-b609-d3c173b27ae5","Type":"ContainerDied","Data":"7452b83aa99dc4b968424e5bfda01396941af2ea71b048aff10757bf5fe809e0"} Nov 03 01:50:49.302879 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:49.302833 5437 scope.go:122] "RemoveContainer" containerID="e37358d2a8a5f7ced043fd9922e3dc4634021ea0e09c7532cd0dd910f4054f0e" Nov 03 01:50:49.318534 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:49.318405 5437 scope.go:122] "RemoveContainer" containerID="e37358d2a8a5f7ced043fd9922e3dc4634021ea0e09c7532cd0dd910f4054f0e" Nov 03 01:50:49.318852 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:50:49.318817 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e37358d2a8a5f7ced043fd9922e3dc4634021ea0e09c7532cd0dd910f4054f0e\": container with ID starting with e37358d2a8a5f7ced043fd9922e3dc4634021ea0e09c7532cd0dd910f4054f0e not found: ID does not exist" containerID="e37358d2a8a5f7ced043fd9922e3dc4634021ea0e09c7532cd0dd910f4054f0e" Nov 03 01:50:49.318932 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:49.318867 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e37358d2a8a5f7ced043fd9922e3dc4634021ea0e09c7532cd0dd910f4054f0e"} err="failed to get container status \"e37358d2a8a5f7ced043fd9922e3dc4634021ea0e09c7532cd0dd910f4054f0e\": rpc error: code = NotFound desc = could not find container \"e37358d2a8a5f7ced043fd9922e3dc4634021ea0e09c7532cd0dd910f4054f0e\": container with ID starting with e37358d2a8a5f7ced043fd9922e3dc4634021ea0e09c7532cd0dd910f4054f0e not found: ID does not exist" Nov 03 01:50:49.474269 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:49.474180 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-jxv68"] Nov 03 01:50:49.477847 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:49.477815 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-jxv68"] Nov 03 01:50:50.090313 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:50.090266 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f118c89-c5d9-4939-b609-d3c173b27ae5" path="/var/lib/kubelet/pods/5f118c89-c5d9-4939-b609-d3c173b27ae5/volumes" Nov 03 01:50:50.876650 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:50.876592 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:50:51.080042 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:51.079982 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:50:51.618713 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:51.618673 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:50:54.502007 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:54.501967 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:50:54.704829 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:54.704793 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:50:54.840745 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:54.840708 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b37ba5-75c4b6866c-tnvxd"] Nov 03 01:50:54.841351 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:54.841333 5437 cpu_manager.go:414] "containerMap: removing container" podUID="5f118c89-c5d9-4939-b609-d3c173b27ae5" containerName="istio-proxy" Nov 03 01:50:54.841423 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:54.841353 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f118c89-c5d9-4939-b609-d3c173b27ae5" containerName="istio-proxy" Nov 03 01:50:54.841483 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:54.841473 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f118c89-c5d9-4939-b609-d3c173b27ae5" containerName="istio-proxy" Nov 03 01:50:54.846438 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:54.846410 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b37ba5-75c4b6866c-tnvxd" Nov 03 01:50:54.900699 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:54.900646 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b37ba5-75c4b6866c-tnvxd"] Nov 03 01:50:54.920167 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:54.920121 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7z2q9\" (UniqueName: \"kubernetes.io/projected/d37572bf-d59f-4e9a-8745-f4d91be336f5-kube-api-access-7z2q9\") pod \"payload-pre-processing-e2e-ipp-b-b37ba5-75c4b6866c-tnvxd\" (UID: \"d37572bf-d59f-4e9a-8745-f4d91be336f5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b37ba5-75c4b6866c-tnvxd" Nov 03 01:50:54.920415 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:54.920342 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d37572bf-d59f-4e9a-8745-f4d91be336f5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b37ba5-75c4b6866c-tnvxd\" (UID: \"d37572bf-d59f-4e9a-8745-f4d91be336f5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b37ba5-75c4b6866c-tnvxd" Nov 03 01:50:54.947887 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:54.947852 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-b37ba5\"" Nov 03 01:50:54.948352 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:54.948283 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-b37ba5-dockercfg-w2p8z\"" Nov 03 01:50:55.007272 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:55.007235 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b37ba5-847f4b6f4c-lt2jx"] Nov 03 01:50:55.013201 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:55.013168 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b37ba5-847f4b6f4c-lt2jx" Nov 03 01:50:55.021417 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:55.021382 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d37572bf-d59f-4e9a-8745-f4d91be336f5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b37ba5-75c4b6866c-tnvxd\" (UID: \"d37572bf-d59f-4e9a-8745-f4d91be336f5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b37ba5-75c4b6866c-tnvxd" Nov 03 01:50:55.021721 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:55.021596 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7z2q9\" (UniqueName: \"kubernetes.io/projected/d37572bf-d59f-4e9a-8745-f4d91be336f5-kube-api-access-7z2q9\") pod \"payload-pre-processing-e2e-ipp-b-b37ba5-75c4b6866c-tnvxd\" (UID: \"d37572bf-d59f-4e9a-8745-f4d91be336f5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b37ba5-75c4b6866c-tnvxd" Nov 03 01:50:55.022338 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:55.022293 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d37572bf-d59f-4e9a-8745-f4d91be336f5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b37ba5-75c4b6866c-tnvxd\" (UID: \"d37572bf-d59f-4e9a-8745-f4d91be336f5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b37ba5-75c4b6866c-tnvxd" Nov 03 01:50:55.066413 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:55.066356 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b37ba5-847f4b6f4c-lt2jx"] Nov 03 01:50:55.088970 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:55.088935 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7z2q9\" (UniqueName: \"kubernetes.io/projected/d37572bf-d59f-4e9a-8745-f4d91be336f5-kube-api-access-7z2q9\") pod \"payload-pre-processing-e2e-ipp-b-b37ba5-75c4b6866c-tnvxd\" (UID: \"d37572bf-d59f-4e9a-8745-f4d91be336f5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b37ba5-75c4b6866c-tnvxd" Nov 03 01:50:55.122827 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:55.122732 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lt657\" (UniqueName: \"kubernetes.io/projected/04cd0e8c-c175-4c88-a5d3-856e5c04fe98-kube-api-access-lt657\") pod \"payload-processing-e2e-ipp-b-b37ba5-847f4b6f4c-lt2jx\" (UID: \"04cd0e8c-c175-4c88-a5d3-856e5c04fe98\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b37ba5-847f4b6f4c-lt2jx" Nov 03 01:50:55.122999 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:55.122875 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04cd0e8c-c175-4c88-a5d3-856e5c04fe98-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b37ba5-847f4b6f4c-lt2jx\" (UID: \"04cd0e8c-c175-4c88-a5d3-856e5c04fe98\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b37ba5-847f4b6f4c-lt2jx" Nov 03 01:50:55.160234 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:55.160194 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b37ba5-75c4b6866c-tnvxd" Nov 03 01:50:55.224765 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:55.224732 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lt657\" (UniqueName: \"kubernetes.io/projected/04cd0e8c-c175-4c88-a5d3-856e5c04fe98-kube-api-access-lt657\") pod \"payload-processing-e2e-ipp-b-b37ba5-847f4b6f4c-lt2jx\" (UID: \"04cd0e8c-c175-4c88-a5d3-856e5c04fe98\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b37ba5-847f4b6f4c-lt2jx" Nov 03 01:50:55.224883 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:55.224824 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04cd0e8c-c175-4c88-a5d3-856e5c04fe98-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b37ba5-847f4b6f4c-lt2jx\" (UID: \"04cd0e8c-c175-4c88-a5d3-856e5c04fe98\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b37ba5-847f4b6f4c-lt2jx" Nov 03 01:50:55.225759 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:55.225731 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04cd0e8c-c175-4c88-a5d3-856e5c04fe98-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b37ba5-847f4b6f4c-lt2jx\" (UID: \"04cd0e8c-c175-4c88-a5d3-856e5c04fe98\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b37ba5-847f4b6f4c-lt2jx" Nov 03 01:50:55.288286 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:55.288243 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lt657\" (UniqueName: \"kubernetes.io/projected/04cd0e8c-c175-4c88-a5d3-856e5c04fe98-kube-api-access-lt657\") pod \"payload-processing-e2e-ipp-b-b37ba5-847f4b6f4c-lt2jx\" (UID: \"04cd0e8c-c175-4c88-a5d3-856e5c04fe98\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b37ba5-847f4b6f4c-lt2jx" Nov 03 01:50:55.330521 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:55.330468 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b37ba5-847f4b6f4c-lt2jx" Nov 03 01:50:55.535846 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:50:55.535813 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd37572bf_d59f_4e9a_8745_f4d91be336f5.slice/crio-c6b22fac60e24fee665f36eb7cdf0e38e07d54016f277214498f1eb3ba7fec42 WatchSource:0}: Error finding container c6b22fac60e24fee665f36eb7cdf0e38e07d54016f277214498f1eb3ba7fec42: Status 404 returned error can't find the container with id c6b22fac60e24fee665f36eb7cdf0e38e07d54016f277214498f1eb3ba7fec42 Nov 03 01:50:55.538814 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:55.538787 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b37ba5-75c4b6866c-tnvxd"] Nov 03 01:50:55.707156 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:50:55.707121 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04cd0e8c_c175_4c88_a5d3_856e5c04fe98.slice/crio-4cb22e6cb5c3541e15ed18301bb93ac98d6f69f8a84e31f4d1b018bf9b775883 WatchSource:0}: Error finding container 4cb22e6cb5c3541e15ed18301bb93ac98d6f69f8a84e31f4d1b018bf9b775883: Status 404 returned error can't find the container with id 4cb22e6cb5c3541e15ed18301bb93ac98d6f69f8a84e31f4d1b018bf9b775883 Nov 03 01:50:55.710012 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:55.709990 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b37ba5-847f4b6f4c-lt2jx"] Nov 03 01:50:56.353547 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:56.353498 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b37ba5-75c4b6866c-tnvxd" event={"ID":"d37572bf-d59f-4e9a-8745-f4d91be336f5","Type":"ContainerStarted","Data":"7f25928b0e73e9f487e73bf4a29769c6ef1c88b7487356095c66e143d3e60688"} Nov 03 01:50:56.353740 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:56.353575 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b37ba5-75c4b6866c-tnvxd" event={"ID":"d37572bf-d59f-4e9a-8745-f4d91be336f5","Type":"ContainerStarted","Data":"c6b22fac60e24fee665f36eb7cdf0e38e07d54016f277214498f1eb3ba7fec42"} Nov 03 01:50:56.353740 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:56.353604 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b37ba5-75c4b6866c-tnvxd" Nov 03 01:50:56.355877 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:56.355836 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b37ba5-847f4b6f4c-lt2jx" event={"ID":"04cd0e8c-c175-4c88-a5d3-856e5c04fe98","Type":"ContainerStarted","Data":"8d0600e8a97ab1a07d92795008375cfc72d2f7981aade7de51624ecb25d3bfff"} Nov 03 01:50:56.356022 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:56.355891 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b37ba5-847f4b6f4c-lt2jx" event={"ID":"04cd0e8c-c175-4c88-a5d3-856e5c04fe98","Type":"ContainerStarted","Data":"4cb22e6cb5c3541e15ed18301bb93ac98d6f69f8a84e31f4d1b018bf9b775883"} Nov 03 01:50:56.356022 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:56.355937 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-b37ba5-847f4b6f4c-lt2jx" Nov 03 01:50:56.474300 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:56.474242 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b37ba5-75c4b6866c-tnvxd" podStartSLOduration=2.474227803 podStartE2EDuration="2.474227803s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:50:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:50:56.472331931 +0000 UTC m=+1938.501814227" watchObservedRunningTime="2026-11-03 01:50:56.474227803 +0000 UTC m=+1938.503710098" Nov 03 01:50:56.591323 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:56.591269 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-b37ba5-847f4b6f4c-lt2jx" podStartSLOduration=2.59125373 podStartE2EDuration="2.59125373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:50:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:50:56.590581216 +0000 UTC m=+1938.620063511" watchObservedRunningTime="2026-11-03 01:50:56.59125373 +0000 UTC m=+1938.620736025" Nov 03 01:50:57.163855 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:50:57.163819 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:03.490052 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:03.489965 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:07.365453 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:07.365421 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b37ba5-75c4b6866c-tnvxd" Nov 03 01:51:07.365842 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:07.365491 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-b37ba5-847f4b6f4c-lt2jx" Nov 03 01:51:08.228043 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:08.228009 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:13.206811 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:13.206772 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:14.035171 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:51:14.035135 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 01:51:16.157800 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:16.157761 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:16.357122 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:16.357086 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:18.045073 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:18.045031 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:18.991846 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:18.991807 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:19.780296 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:19.780260 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:20.140138 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:20.140097 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:20.493719 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:20.493629 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:22.803407 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:22.803337 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xb7x2"] Nov 03 01:51:22.803883 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:22.803621 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xb7x2" podUID="4b4a66c8-89c8-4894-9244-2a0c3587a06c" containerName="istio-proxy" containerID="cri-o://7135cfd55c1c212acf25acdbe635f8de4242b4df092daf702f82eb40fe4ac803" gracePeriod=30 Nov 03 01:51:22.854548 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:22.854504 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hhnxh"] Nov 03 01:51:22.854752 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:22.854722 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hhnxh" podUID="978393a8-09d7-48b6-9104-d1cae96b8f18" containerName="istio-proxy" containerID="cri-o://e7532f72870ccc430bd769ccf699ef97fc9d6165591283ddfa39a1d721cbd687" gracePeriod=30 Nov 03 01:51:24.602569 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:24.602525 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:24.723006 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:24.722957 5437 patch_prober.go:29] interesting pod/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hhnxh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 01:51:24.723189 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:24.723047 5437 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hhnxh" podUID="978393a8-09d7-48b6-9104-d1cae96b8f18" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 01:51:24.860905 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:24.860807 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:27.703044 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:27.703003 5437 patch_prober.go:29] interesting pod/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xb7x2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 01:51:27.703484 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:27.703064 5437 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xb7x2" podUID="4b4a66c8-89c8-4894-9244-2a0c3587a06c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 01:51:27.751476 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:27.751428 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:27.951007 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:27.950968 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:28.107732 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:28.105601 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0e0d61-6d6d98697f-x9w8p"] Nov 03 01:51:28.109924 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:28.109888 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0e0d61-6d6d98697f-x9w8p" Nov 03 01:51:28.164838 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:28.164806 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0e0d61-6d6d98697f-x9w8p"] Nov 03 01:51:28.180699 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:28.180664 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hhnxh" Nov 03 01:51:28.213264 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:28.213171 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xb7x2" Nov 03 01:51:28.213497 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:28.213382 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-0e0d61\"" Nov 03 01:51:28.213577 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:28.213501 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-0e0d61-dockercfg-vvbls\"" Nov 03 01:51:28.263047 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:28.262718 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/978393a8-09d7-48b6-9104-d1cae96b8f18-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/978393a8-09d7-48b6-9104-d1cae96b8f18-istio-podinfo\") pod \"978393a8-09d7-48b6-9104-d1cae96b8f18\" (UID: \"978393a8-09d7-48b6-9104-d1cae96b8f18\") " Nov 03 01:51:28.263047 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:28.262813 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/978393a8-09d7-48b6-9104-d1cae96b8f18-istio-token\" (UniqueName: \"kubernetes.io/projected/978393a8-09d7-48b6-9104-d1cae96b8f18-istio-token\") pod \"978393a8-09d7-48b6-9104-d1cae96b8f18\" (UID: \"978393a8-09d7-48b6-9104-d1cae96b8f18\") " Nov 03 01:51:28.263047 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:28.262852 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/978393a8-09d7-48b6-9104-d1cae96b8f18-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/978393a8-09d7-48b6-9104-d1cae96b8f18-istiod-ca-cert\") pod \"978393a8-09d7-48b6-9104-d1cae96b8f18\" (UID: \"978393a8-09d7-48b6-9104-d1cae96b8f18\") " Nov 03 01:51:28.263047 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:28.262900 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/978393a8-09d7-48b6-9104-d1cae96b8f18-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/978393a8-09d7-48b6-9104-d1cae96b8f18-credential-socket\") pod \"978393a8-09d7-48b6-9104-d1cae96b8f18\" (UID: \"978393a8-09d7-48b6-9104-d1cae96b8f18\") " Nov 03 01:51:28.263047 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:28.262976 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/978393a8-09d7-48b6-9104-d1cae96b8f18-kube-api-access-jxtmb\" (UniqueName: \"kubernetes.io/projected/978393a8-09d7-48b6-9104-d1cae96b8f18-kube-api-access-jxtmb\") pod \"978393a8-09d7-48b6-9104-d1cae96b8f18\" (UID: \"978393a8-09d7-48b6-9104-d1cae96b8f18\") " Nov 03 01:51:28.263560 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:28.263463 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/978393a8-09d7-48b6-9104-d1cae96b8f18-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/978393a8-09d7-48b6-9104-d1cae96b8f18-istio-data\") pod \"978393a8-09d7-48b6-9104-d1cae96b8f18\" (UID: \"978393a8-09d7-48b6-9104-d1cae96b8f18\") " Nov 03 01:51:28.263560 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:28.263503 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/978393a8-09d7-48b6-9104-d1cae96b8f18-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/978393a8-09d7-48b6-9104-d1cae96b8f18-workload-socket\") pod \"978393a8-09d7-48b6-9104-d1cae96b8f18\" (UID: \"978393a8-09d7-48b6-9104-d1cae96b8f18\") " Nov 03 01:51:28.263674 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:28.263603 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/978393a8-09d7-48b6-9104-d1cae96b8f18-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/978393a8-09d7-48b6-9104-d1cae96b8f18-istio-envoy\") pod \"978393a8-09d7-48b6-9104-d1cae96b8f18\" (UID: \"978393a8-09d7-48b6-9104-d1cae96b8f18\") " Nov 03 01:51:28.263735 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:28.263720 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/978393a8-09d7-48b6-9104-d1cae96b8f18-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/978393a8-09d7-48b6-9104-d1cae96b8f18-workload-certs\") pod \"978393a8-09d7-48b6-9104-d1cae96b8f18\" (UID: \"978393a8-09d7-48b6-9104-d1cae96b8f18\") " Nov 03 01:51:28.263930 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:28.263911 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ck6w2\" (UniqueName: \"kubernetes.io/projected/43733d32-180d-4918-a97e-0e36ac3405da-kube-api-access-ck6w2\") pod \"payload-pre-processing-e2e-api-b-0e0d61-6d6d98697f-x9w8p\" (UID: \"43733d32-180d-4918-a97e-0e36ac3405da\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0e0d61-6d6d98697f-x9w8p" Nov 03 01:51:28.264023 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:28.263980 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43733d32-180d-4918-a97e-0e36ac3405da-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-0e0d61-6d6d98697f-x9w8p\" (UID: \"43733d32-180d-4918-a97e-0e36ac3405da\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0e0d61-6d6d98697f-x9w8p" Nov 03 01:51:28.265135 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:28.264778 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/978393a8-09d7-48b6-9104-d1cae96b8f18-workload-certs" pod "978393a8-09d7-48b6-9104-d1cae96b8f18" (UID: "978393a8-09d7-48b6-9104-d1cae96b8f18"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:51:28.265135 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:28.265077 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/978393a8-09d7-48b6-9104-d1cae96b8f18-workload-socket" pod "978393a8-09d7-48b6-9104-d1cae96b8f18" (UID: "978393a8-09d7-48b6-9104-d1cae96b8f18"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:51:28.265135 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:28.265093 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/978393a8-09d7-48b6-9104-d1cae96b8f18-credential-socket" pod "978393a8-09d7-48b6-9104-d1cae96b8f18" (UID: "978393a8-09d7-48b6-9104-d1cae96b8f18"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:51:28.265390 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:28.265265 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/978393a8-09d7-48b6-9104-d1cae96b8f18-istiod-ca-cert" pod "978393a8-09d7-48b6-9104-d1cae96b8f18" (UID: "978393a8-09d7-48b6-9104-d1cae96b8f18"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:51:28.265942 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:28.265907 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/978393a8-09d7-48b6-9104-d1cae96b8f18-istio-data" pod "978393a8-09d7-48b6-9104-d1cae96b8f18" (UID: "978393a8-09d7-48b6-9104-d1cae96b8f18"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:51:28.272116 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:28.272079 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/978393a8-09d7-48b6-9104-d1cae96b8f18-istio-token" pod "978393a8-09d7-48b6-9104-d1cae96b8f18" (UID: "978393a8-09d7-48b6-9104-d1cae96b8f18"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:51:28.274979 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:28.274943 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/978393a8-09d7-48b6-9104-d1cae96b8f18-istio-envoy" pod "978393a8-09d7-48b6-9104-d1cae96b8f18" (UID: "978393a8-09d7-48b6-9104-d1cae96b8f18"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:51:28.275213 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:28.275105 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/978393a8-09d7-48b6-9104-d1cae96b8f18-kube-api-access-jxtmb" pod "978393a8-09d7-48b6-9104-d1cae96b8f18" (UID: "978393a8-09d7-48b6-9104-d1cae96b8f18"). InnerVolumeSpecName "kube-api-access-jxtmb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:51:28.299653 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:28.299597 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/978393a8-09d7-48b6-9104-d1cae96b8f18-istio-podinfo" pod "978393a8-09d7-48b6-9104-d1cae96b8f18" (UID: "978393a8-09d7-48b6-9104-d1cae96b8f18"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:51:28.364715 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:28.364681 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b4a66c8-89c8-4894-9244-2a0c3587a06c-istio-token\" (UniqueName: \"kubernetes.io/projected/4b4a66c8-89c8-4894-9244-2a0c3587a06c-istio-token\") pod \"4b4a66c8-89c8-4894-9244-2a0c3587a06c\" (UID: \"4b4a66c8-89c8-4894-9244-2a0c3587a06c\") " Nov 03 01:51:28.364912 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:28.364728 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b4a66c8-89c8-4894-9244-2a0c3587a06c-kube-api-access-7srjg\" (UniqueName: \"kubernetes.io/projected/4b4a66c8-89c8-4894-9244-2a0c3587a06c-kube-api-access-7srjg\") pod \"4b4a66c8-89c8-4894-9244-2a0c3587a06c\" (UID: \"4b4a66c8-89c8-4894-9244-2a0c3587a06c\") " Nov 03 01:51:28.364912 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:28.364752 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b4a66c8-89c8-4894-9244-2a0c3587a06c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b4a66c8-89c8-4894-9244-2a0c3587a06c-istio-data\") pod \"4b4a66c8-89c8-4894-9244-2a0c3587a06c\" (UID: \"4b4a66c8-89c8-4894-9244-2a0c3587a06c\") " Nov 03 01:51:28.364912 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:28.364859 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b4a66c8-89c8-4894-9244-2a0c3587a06c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b4a66c8-89c8-4894-9244-2a0c3587a06c-workload-socket\") pod \"4b4a66c8-89c8-4894-9244-2a0c3587a06c\" (UID: \"4b4a66c8-89c8-4894-9244-2a0c3587a06c\") " Nov 03 01:51:28.365082 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:28.365039 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b4a66c8-89c8-4894-9244-2a0c3587a06c-istio-data" pod "4b4a66c8-89c8-4894-9244-2a0c3587a06c" (UID: "4b4a66c8-89c8-4894-9244-2a0c3587a06c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:51:28.365169 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:28.365115 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b4a66c8-89c8-4894-9244-2a0c3587a06c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b4a66c8-89c8-4894-9244-2a0c3587a06c-workload-certs\") pod \"4b4a66c8-89c8-4894-9244-2a0c3587a06c\" (UID: \"4b4a66c8-89c8-4894-9244-2a0c3587a06c\") " Nov 03 01:51:28.365223 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:28.365181 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b4a66c8-89c8-4894-9244-2a0c3587a06c-workload-socket" pod "4b4a66c8-89c8-4894-9244-2a0c3587a06c" (UID: "4b4a66c8-89c8-4894-9244-2a0c3587a06c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:51:28.365223 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:28.365207 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b4a66c8-89c8-4894-9244-2a0c3587a06c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b4a66c8-89c8-4894-9244-2a0c3587a06c-istiod-ca-cert\") pod \"4b4a66c8-89c8-4894-9244-2a0c3587a06c\" (UID: \"4b4a66c8-89c8-4894-9244-2a0c3587a06c\") " Nov 03 01:51:28.365322 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:28.365288 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4b4a66c8-89c8-4894-9244-2a0c3587a06c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b4a66c8-89c8-4894-9244-2a0c3587a06c-istio-podinfo\") pod \"4b4a66c8-89c8-4894-9244-2a0c3587a06c\" (UID: \"4b4a66c8-89c8-4894-9244-2a0c3587a06c\") " Nov 03 01:51:28.365378 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:28.365340 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b4a66c8-89c8-4894-9244-2a0c3587a06c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b4a66c8-89c8-4894-9244-2a0c3587a06c-credential-socket\") pod \"4b4a66c8-89c8-4894-9244-2a0c3587a06c\" (UID: \"4b4a66c8-89c8-4894-9244-2a0c3587a06c\") " Nov 03 01:51:28.365437 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:28.365402 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b4a66c8-89c8-4894-9244-2a0c3587a06c-workload-certs" pod "4b4a66c8-89c8-4894-9244-2a0c3587a06c" (UID: "4b4a66c8-89c8-4894-9244-2a0c3587a06c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:51:28.365437 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:28.365408 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b4a66c8-89c8-4894-9244-2a0c3587a06c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b4a66c8-89c8-4894-9244-2a0c3587a06c-istio-envoy\") pod \"4b4a66c8-89c8-4894-9244-2a0c3587a06c\" (UID: \"4b4a66c8-89c8-4894-9244-2a0c3587a06c\") " Nov 03 01:51:28.365633 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:28.365606 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b4a66c8-89c8-4894-9244-2a0c3587a06c-istiod-ca-cert" pod "4b4a66c8-89c8-4894-9244-2a0c3587a06c" (UID: "4b4a66c8-89c8-4894-9244-2a0c3587a06c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:51:28.365701 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:28.365635 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b4a66c8-89c8-4894-9244-2a0c3587a06c-credential-socket" pod "4b4a66c8-89c8-4894-9244-2a0c3587a06c" (UID: "4b4a66c8-89c8-4894-9244-2a0c3587a06c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:51:28.365778 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:28.365750 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43733d32-180d-4918-a97e-0e36ac3405da-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-0e0d61-6d6d98697f-x9w8p\" (UID: \"43733d32-180d-4918-a97e-0e36ac3405da\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0e0d61-6d6d98697f-x9w8p" Nov 03 01:51:28.366016 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:28.365999 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ck6w2\" (UniqueName: \"kubernetes.io/projected/43733d32-180d-4918-a97e-0e36ac3405da-kube-api-access-ck6w2\") pod \"payload-pre-processing-e2e-api-b-0e0d61-6d6d98697f-x9w8p\" (UID: \"43733d32-180d-4918-a97e-0e36ac3405da\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0e0d61-6d6d98697f-x9w8p" Nov 03 01:51:28.366093 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:28.366084 5437 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/978393a8-09d7-48b6-9104-d1cae96b8f18-workload-certs\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:51:28.366150 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:28.366103 5437 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/978393a8-09d7-48b6-9104-d1cae96b8f18-istio-podinfo\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:51:28.366150 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:28.366120 5437 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b4a66c8-89c8-4894-9244-2a0c3587a06c-workload-certs\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:51:28.366150 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:28.366134 5437 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/978393a8-09d7-48b6-9104-d1cae96b8f18-istio-token\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:51:28.366319 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:28.366148 5437 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/978393a8-09d7-48b6-9104-d1cae96b8f18-istiod-ca-cert\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:51:28.366319 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:28.366162 5437 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b4a66c8-89c8-4894-9244-2a0c3587a06c-istiod-ca-cert\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:51:28.366319 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:28.366177 5437 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/978393a8-09d7-48b6-9104-d1cae96b8f18-credential-socket\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:51:28.366319 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:28.366191 5437 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b4a66c8-89c8-4894-9244-2a0c3587a06c-credential-socket\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:51:28.366319 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:28.366206 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxtmb\" (UniqueName: \"kubernetes.io/projected/978393a8-09d7-48b6-9104-d1cae96b8f18-kube-api-access-jxtmb\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:51:28.366319 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:28.366219 5437 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b4a66c8-89c8-4894-9244-2a0c3587a06c-istio-data\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:51:28.366319 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:28.366232 5437 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b4a66c8-89c8-4894-9244-2a0c3587a06c-workload-socket\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:51:28.366319 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:28.366247 5437 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/978393a8-09d7-48b6-9104-d1cae96b8f18-istio-data\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:51:28.366319 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:28.366263 5437 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/978393a8-09d7-48b6-9104-d1cae96b8f18-workload-socket\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:51:28.366319 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:28.366277 5437 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/978393a8-09d7-48b6-9104-d1cae96b8f18-istio-envoy\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:51:28.366809 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:28.366701 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43733d32-180d-4918-a97e-0e36ac3405da-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-0e0d61-6d6d98697f-x9w8p\" (UID: \"43733d32-180d-4918-a97e-0e36ac3405da\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0e0d61-6d6d98697f-x9w8p" Nov 03 01:51:28.367739 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:28.367701 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b4a66c8-89c8-4894-9244-2a0c3587a06c-kube-api-access-7srjg" pod "4b4a66c8-89c8-4894-9244-2a0c3587a06c" (UID: "4b4a66c8-89c8-4894-9244-2a0c3587a06c"). InnerVolumeSpecName "kube-api-access-7srjg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:51:28.367843 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:28.367784 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b4a66c8-89c8-4894-9244-2a0c3587a06c-istio-envoy" pod "4b4a66c8-89c8-4894-9244-2a0c3587a06c" (UID: "4b4a66c8-89c8-4894-9244-2a0c3587a06c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:51:28.368441 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:28.368419 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4b4a66c8-89c8-4894-9244-2a0c3587a06c-istio-podinfo" pod "4b4a66c8-89c8-4894-9244-2a0c3587a06c" (UID: "4b4a66c8-89c8-4894-9244-2a0c3587a06c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:51:28.369102 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:28.369083 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b4a66c8-89c8-4894-9244-2a0c3587a06c-istio-token" pod "4b4a66c8-89c8-4894-9244-2a0c3587a06c" (UID: "4b4a66c8-89c8-4894-9244-2a0c3587a06c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:51:28.431791 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:28.431714 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ck6w2\" (UniqueName: \"kubernetes.io/projected/43733d32-180d-4918-a97e-0e36ac3405da-kube-api-access-ck6w2\") pod \"payload-pre-processing-e2e-api-b-0e0d61-6d6d98697f-x9w8p\" (UID: \"43733d32-180d-4918-a97e-0e36ac3405da\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0e0d61-6d6d98697f-x9w8p" Nov 03 01:51:28.433413 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:28.433387 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0e0d61-6d6d98697f-x9w8p" Nov 03 01:51:28.467456 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:28.467338 5437 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b4a66c8-89c8-4894-9244-2a0c3587a06c-istio-podinfo\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:51:28.467456 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:28.467410 5437 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b4a66c8-89c8-4894-9244-2a0c3587a06c-istio-envoy\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:51:28.467456 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:28.467424 5437 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b4a66c8-89c8-4894-9244-2a0c3587a06c-istio-token\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:51:28.467456 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:28.467438 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7srjg\" (UniqueName: \"kubernetes.io/projected/4b4a66c8-89c8-4894-9244-2a0c3587a06c-kube-api-access-7srjg\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:51:28.575680 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:28.575640 5437 generic.go:356] "Generic (PLEG): container finished" podID="978393a8-09d7-48b6-9104-d1cae96b8f18" containerID="e7532f72870ccc430bd769ccf699ef97fc9d6165591283ddfa39a1d721cbd687" exitCode=0 Nov 03 01:51:28.575680 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:28.575692 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hhnxh" Nov 03 01:51:28.575959 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:28.575719 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hhnxh" event={"ID":"978393a8-09d7-48b6-9104-d1cae96b8f18","Type":"ContainerDied","Data":"e7532f72870ccc430bd769ccf699ef97fc9d6165591283ddfa39a1d721cbd687"} Nov 03 01:51:28.575959 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:28.575780 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hhnxh" event={"ID":"978393a8-09d7-48b6-9104-d1cae96b8f18","Type":"ContainerDied","Data":"ca4fe70661f4d83d04bc51be3877b133d9a1be4c04b7cd3131b34e40cd49f0d4"} Nov 03 01:51:28.575959 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:28.575807 5437 scope.go:122] "RemoveContainer" containerID="e7532f72870ccc430bd769ccf699ef97fc9d6165591283ddfa39a1d721cbd687" Nov 03 01:51:28.577588 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:28.577559 5437 generic.go:356] "Generic (PLEG): container finished" podID="4b4a66c8-89c8-4894-9244-2a0c3587a06c" containerID="7135cfd55c1c212acf25acdbe635f8de4242b4df092daf702f82eb40fe4ac803" exitCode=0 Nov 03 01:51:28.577725 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:28.577630 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xb7x2" event={"ID":"4b4a66c8-89c8-4894-9244-2a0c3587a06c","Type":"ContainerDied","Data":"7135cfd55c1c212acf25acdbe635f8de4242b4df092daf702f82eb40fe4ac803"} Nov 03 01:51:28.577725 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:28.577675 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xb7x2" event={"ID":"4b4a66c8-89c8-4894-9244-2a0c3587a06c","Type":"ContainerDied","Data":"ab06b1f56962851c9432108a755020631ba9611a982c469c52f22c1e5013b888"} Nov 03 01:51:28.577725 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:28.577719 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xb7x2" Nov 03 01:51:28.588913 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:28.588885 5437 scope.go:122] "RemoveContainer" containerID="e7532f72870ccc430bd769ccf699ef97fc9d6165591283ddfa39a1d721cbd687" Nov 03 01:51:28.589233 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:51:28.589210 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7532f72870ccc430bd769ccf699ef97fc9d6165591283ddfa39a1d721cbd687\": container with ID starting with e7532f72870ccc430bd769ccf699ef97fc9d6165591283ddfa39a1d721cbd687 not found: ID does not exist" containerID="e7532f72870ccc430bd769ccf699ef97fc9d6165591283ddfa39a1d721cbd687" Nov 03 01:51:28.589292 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:28.589242 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7532f72870ccc430bd769ccf699ef97fc9d6165591283ddfa39a1d721cbd687"} err="failed to get container status \"e7532f72870ccc430bd769ccf699ef97fc9d6165591283ddfa39a1d721cbd687\": rpc error: code = NotFound desc = could not find container \"e7532f72870ccc430bd769ccf699ef97fc9d6165591283ddfa39a1d721cbd687\": container with ID starting with e7532f72870ccc430bd769ccf699ef97fc9d6165591283ddfa39a1d721cbd687 not found: ID does not exist" Nov 03 01:51:28.589292 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:28.589260 5437 scope.go:122] "RemoveContainer" containerID="7135cfd55c1c212acf25acdbe635f8de4242b4df092daf702f82eb40fe4ac803" Nov 03 01:51:28.599031 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:28.599006 5437 scope.go:122] "RemoveContainer" containerID="7135cfd55c1c212acf25acdbe635f8de4242b4df092daf702f82eb40fe4ac803" Nov 03 01:51:28.599350 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:51:28.599333 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7135cfd55c1c212acf25acdbe635f8de4242b4df092daf702f82eb40fe4ac803\": container with ID starting with 7135cfd55c1c212acf25acdbe635f8de4242b4df092daf702f82eb40fe4ac803 not found: ID does not exist" containerID="7135cfd55c1c212acf25acdbe635f8de4242b4df092daf702f82eb40fe4ac803" Nov 03 01:51:28.599432 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:28.599359 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7135cfd55c1c212acf25acdbe635f8de4242b4df092daf702f82eb40fe4ac803"} err="failed to get container status \"7135cfd55c1c212acf25acdbe635f8de4242b4df092daf702f82eb40fe4ac803\": rpc error: code = NotFound desc = could not find container \"7135cfd55c1c212acf25acdbe635f8de4242b4df092daf702f82eb40fe4ac803\": container with ID starting with 7135cfd55c1c212acf25acdbe635f8de4242b4df092daf702f82eb40fe4ac803 not found: ID does not exist" Nov 03 01:51:28.748715 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:28.748605 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hhnxh"] Nov 03 01:51:28.754222 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:28.754187 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hhnxh"] Nov 03 01:51:28.849206 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:51:28.849171 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43733d32_180d_4918_a97e_0e36ac3405da.slice/crio-36fd2fb2abb564b56aecc7626028842ff9b7f1b5ea890880fb20aa4383e7a5ca WatchSource:0}: Error finding container 36fd2fb2abb564b56aecc7626028842ff9b7f1b5ea890880fb20aa4383e7a5ca: Status 404 returned error can't find the container with id 36fd2fb2abb564b56aecc7626028842ff9b7f1b5ea890880fb20aa4383e7a5ca Nov 03 01:51:28.851839 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:28.851815 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0e0d61-6d6d98697f-x9w8p"] Nov 03 01:51:28.865894 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:28.865860 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xb7x2"] Nov 03 01:51:28.870899 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:28.870869 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xb7x2"] Nov 03 01:51:29.586798 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:29.586751 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0e0d61-6d6d98697f-x9w8p" event={"ID":"43733d32-180d-4918-a97e-0e36ac3405da","Type":"ContainerStarted","Data":"b2eb1ee052c6e17fc026c9c31745407ec2b1e92038edf8c30c6cd9f45014bbe9"} Nov 03 01:51:29.587009 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:29.586837 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0e0d61-6d6d98697f-x9w8p" Nov 03 01:51:29.587009 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:29.586865 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0e0d61-6d6d98697f-x9w8p" event={"ID":"43733d32-180d-4918-a97e-0e36ac3405da","Type":"ContainerStarted","Data":"36fd2fb2abb564b56aecc7626028842ff9b7f1b5ea890880fb20aa4383e7a5ca"} Nov 03 01:51:29.615340 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:29.615301 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-71746a-775978fc79-vbnnk"] Nov 03 01:51:29.615955 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:29.615937 5437 cpu_manager.go:414] "containerMap: removing container" podUID="4b4a66c8-89c8-4894-9244-2a0c3587a06c" containerName="istio-proxy" Nov 03 01:51:29.615996 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:29.615956 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b4a66c8-89c8-4894-9244-2a0c3587a06c" containerName="istio-proxy" Nov 03 01:51:29.615996 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:29.615984 5437 cpu_manager.go:414] "containerMap: removing container" podUID="978393a8-09d7-48b6-9104-d1cae96b8f18" containerName="istio-proxy" Nov 03 01:51:29.615996 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:29.615989 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="978393a8-09d7-48b6-9104-d1cae96b8f18" containerName="istio-proxy" Nov 03 01:51:29.616092 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:29.616072 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b4a66c8-89c8-4894-9244-2a0c3587a06c" containerName="istio-proxy" Nov 03 01:51:29.616092 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:29.616083 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="978393a8-09d7-48b6-9104-d1cae96b8f18" containerName="istio-proxy" Nov 03 01:51:29.619582 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:29.619554 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-71746a-775978fc79-vbnnk" Nov 03 01:51:29.673858 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:29.673821 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-71746a-serving-cert\"" Nov 03 01:51:29.674057 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:29.673990 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-71746a-775978fc79-vbnnk"] Nov 03 01:51:29.710608 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:29.710544 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0e0d61-6d6d98697f-x9w8p" podStartSLOduration=1.710522773 podStartE2EDuration="1.710522773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:51:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:51:29.707122822 +0000 UTC m=+1971.736605122" watchObservedRunningTime="2026-11-03 01:51:29.710522773 +0000 UTC m=+1971.740005066" Nov 03 01:51:29.784871 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:29.784834 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwn5r\" (UniqueName: \"kubernetes.io/projected/9a38ff04-2bef-4dc5-8e95-ae1356bf16f4-kube-api-access-nwn5r\") pod \"maas-api-e2e-ipp-b-71746a-775978fc79-vbnnk\" (UID: \"9a38ff04-2bef-4dc5-8e95-ae1356bf16f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-71746a-775978fc79-vbnnk" Nov 03 01:51:29.785348 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:29.784903 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a38ff04-2bef-4dc5-8e95-ae1356bf16f4-maas-api-tls\") pod \"maas-api-e2e-ipp-b-71746a-775978fc79-vbnnk\" (UID: \"9a38ff04-2bef-4dc5-8e95-ae1356bf16f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-71746a-775978fc79-vbnnk" Nov 03 01:51:29.785348 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:29.784998 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a38ff04-2bef-4dc5-8e95-ae1356bf16f4-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-71746a-775978fc79-vbnnk\" (UID: \"9a38ff04-2bef-4dc5-8e95-ae1356bf16f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-71746a-775978fc79-vbnnk" Nov 03 01:51:29.785348 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:29.785195 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a38ff04-2bef-4dc5-8e95-ae1356bf16f4-metrics-certs\") pod \"maas-api-e2e-ipp-b-71746a-775978fc79-vbnnk\" (UID: \"9a38ff04-2bef-4dc5-8e95-ae1356bf16f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-71746a-775978fc79-vbnnk" Nov 03 01:51:29.849196 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:29.844751 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:29.886017 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:29.885973 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a38ff04-2bef-4dc5-8e95-ae1356bf16f4-metrics-certs\") pod \"maas-api-e2e-ipp-b-71746a-775978fc79-vbnnk\" (UID: \"9a38ff04-2bef-4dc5-8e95-ae1356bf16f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-71746a-775978fc79-vbnnk" Nov 03 01:51:29.886220 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:29.886042 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwn5r\" (UniqueName: \"kubernetes.io/projected/9a38ff04-2bef-4dc5-8e95-ae1356bf16f4-kube-api-access-nwn5r\") pod \"maas-api-e2e-ipp-b-71746a-775978fc79-vbnnk\" (UID: \"9a38ff04-2bef-4dc5-8e95-ae1356bf16f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-71746a-775978fc79-vbnnk" Nov 03 01:51:29.886220 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:29.886083 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a38ff04-2bef-4dc5-8e95-ae1356bf16f4-maas-api-tls\") pod \"maas-api-e2e-ipp-b-71746a-775978fc79-vbnnk\" (UID: \"9a38ff04-2bef-4dc5-8e95-ae1356bf16f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-71746a-775978fc79-vbnnk" Nov 03 01:51:29.886220 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:29.886109 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a38ff04-2bef-4dc5-8e95-ae1356bf16f4-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-71746a-775978fc79-vbnnk\" (UID: \"9a38ff04-2bef-4dc5-8e95-ae1356bf16f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-71746a-775978fc79-vbnnk" Nov 03 01:51:29.886932 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:29.886908 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a38ff04-2bef-4dc5-8e95-ae1356bf16f4-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-71746a-775978fc79-vbnnk\" (UID: \"9a38ff04-2bef-4dc5-8e95-ae1356bf16f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-71746a-775978fc79-vbnnk" Nov 03 01:51:29.888715 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:29.888684 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a38ff04-2bef-4dc5-8e95-ae1356bf16f4-metrics-certs\") pod \"maas-api-e2e-ipp-b-71746a-775978fc79-vbnnk\" (UID: \"9a38ff04-2bef-4dc5-8e95-ae1356bf16f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-71746a-775978fc79-vbnnk" Nov 03 01:51:29.888826 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:29.888684 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a38ff04-2bef-4dc5-8e95-ae1356bf16f4-maas-api-tls\") pod \"maas-api-e2e-ipp-b-71746a-775978fc79-vbnnk\" (UID: \"9a38ff04-2bef-4dc5-8e95-ae1356bf16f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-71746a-775978fc79-vbnnk" Nov 03 01:51:29.949397 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:29.949348 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwn5r\" (UniqueName: \"kubernetes.io/projected/9a38ff04-2bef-4dc5-8e95-ae1356bf16f4-kube-api-access-nwn5r\") pod \"maas-api-e2e-ipp-b-71746a-775978fc79-vbnnk\" (UID: \"9a38ff04-2bef-4dc5-8e95-ae1356bf16f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-71746a-775978fc79-vbnnk" Nov 03 01:51:30.051881 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:30.051842 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:30.089429 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:30.089392 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b4a66c8-89c8-4894-9244-2a0c3587a06c" path="/var/lib/kubelet/pods/4b4a66c8-89c8-4894-9244-2a0c3587a06c/volumes" Nov 03 01:51:30.089901 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:30.089885 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="978393a8-09d7-48b6-9104-d1cae96b8f18" path="/var/lib/kubelet/pods/978393a8-09d7-48b6-9104-d1cae96b8f18/volumes" Nov 03 01:51:30.234042 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:30.233946 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-71746a-775978fc79-vbnnk" Nov 03 01:51:30.255780 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:30.255635 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:30.613315 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:51:30.613284 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a38ff04_2bef_4dc5_8e95_ae1356bf16f4.slice/crio-4fe272b4a3d4100bd2d7bbf620eea1b2dcb1f259b6d96f1276097d8c0115775f WatchSource:0}: Error finding container 4fe272b4a3d4100bd2d7bbf620eea1b2dcb1f259b6d96f1276097d8c0115775f: Status 404 returned error can't find the container with id 4fe272b4a3d4100bd2d7bbf620eea1b2dcb1f259b6d96f1276097d8c0115775f Nov 03 01:51:30.615052 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:30.615030 5437 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 01:51:30.616574 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:30.616549 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-71746a-775978fc79-vbnnk"] Nov 03 01:51:31.602996 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:31.602949 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-71746a-775978fc79-vbnnk" event={"ID":"9a38ff04-2bef-4dc5-8e95-ae1356bf16f4","Type":"ContainerStarted","Data":"fce255a63795568ea97fb41c10b880bb5e0ccbd016dd91d55716851c77aff336"} Nov 03 01:51:31.603452 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:31.603020 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-71746a-775978fc79-vbnnk" event={"ID":"9a38ff04-2bef-4dc5-8e95-ae1356bf16f4","Type":"ContainerStarted","Data":"4fe272b4a3d4100bd2d7bbf620eea1b2dcb1f259b6d96f1276097d8c0115775f"} Nov 03 01:51:31.603452 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:31.603154 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-71746a-775978fc79-vbnnk" Nov 03 01:51:31.759944 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:31.759830 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-71746a-775978fc79-vbnnk" podStartSLOduration=1.854096442 podStartE2EDuration="2.75980654s" totalImagesPullingTime="905.710098ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:51:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:51:31.75751643 +0000 UTC m=+1973.786998741" watchObservedRunningTime="2026-11-03 01:51:31.75980654 +0000 UTC m=+1973.789288837" Nov 03 01:51:35.196293 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:35.196253 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:37.615646 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:37.615616 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-71746a-775978fc79-vbnnk" Nov 03 01:51:40.182641 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:40.182573 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:40.598285 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:40.598249 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0e0d61-6d6d98697f-x9w8p" Nov 03 01:51:44.037419 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:51:44.037359 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 01:51:44.728740 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:44.728705 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:44.986043 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:44.985941 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:45.429956 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:45.429920 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:46.963957 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:46.963869 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:48.837110 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:48.837058 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:49.303875 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:49.303784 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:51.976804 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:51.976766 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:56.691586 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:51:56.691545 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:52:01.062650 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:52:01.062608 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b37ba5-d668d8449-xqgrq"] Nov 03 01:52:01.066602 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:52:01.066578 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b37ba5-d668d8449-xqgrq" Nov 03 01:52:01.099600 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:52:01.099564 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8282fcb6-7612-4ade-9386-3b1e7ae11b13-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-b37ba5-d668d8449-xqgrq\" (UID: \"8282fcb6-7612-4ade-9386-3b1e7ae11b13\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b37ba5-d668d8449-xqgrq" Nov 03 01:52:01.099769 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:52:01.099738 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8282fcb6-7612-4ade-9386-3b1e7ae11b13-metrics-certs\") pod \"maas-api-e2e-ipp-b-b37ba5-d668d8449-xqgrq\" (UID: \"8282fcb6-7612-4ade-9386-3b1e7ae11b13\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b37ba5-d668d8449-xqgrq" Nov 03 01:52:01.099816 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:52:01.099770 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8282fcb6-7612-4ade-9386-3b1e7ae11b13-maas-api-tls\") pod \"maas-api-e2e-ipp-b-b37ba5-d668d8449-xqgrq\" (UID: \"8282fcb6-7612-4ade-9386-3b1e7ae11b13\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b37ba5-d668d8449-xqgrq" Nov 03 01:52:01.099852 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:52:01.099829 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnzft\" (UniqueName: \"kubernetes.io/projected/8282fcb6-7612-4ade-9386-3b1e7ae11b13-kube-api-access-pnzft\") pod \"maas-api-e2e-ipp-b-b37ba5-d668d8449-xqgrq\" (UID: \"8282fcb6-7612-4ade-9386-3b1e7ae11b13\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b37ba5-d668d8449-xqgrq" Nov 03 01:52:01.121775 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:52:01.121739 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b37ba5-d668d8449-xqgrq"] Nov 03 01:52:01.170477 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:52:01.170444 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-b37ba5-serving-cert\"" Nov 03 01:52:01.201250 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:52:01.201211 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8282fcb6-7612-4ade-9386-3b1e7ae11b13-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-b37ba5-d668d8449-xqgrq\" (UID: \"8282fcb6-7612-4ade-9386-3b1e7ae11b13\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b37ba5-d668d8449-xqgrq" Nov 03 01:52:01.201470 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:52:01.201412 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8282fcb6-7612-4ade-9386-3b1e7ae11b13-metrics-certs\") pod \"maas-api-e2e-ipp-b-b37ba5-d668d8449-xqgrq\" (UID: \"8282fcb6-7612-4ade-9386-3b1e7ae11b13\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b37ba5-d668d8449-xqgrq" Nov 03 01:52:01.201470 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:52:01.201446 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8282fcb6-7612-4ade-9386-3b1e7ae11b13-maas-api-tls\") pod \"maas-api-e2e-ipp-b-b37ba5-d668d8449-xqgrq\" (UID: \"8282fcb6-7612-4ade-9386-3b1e7ae11b13\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b37ba5-d668d8449-xqgrq" Nov 03 01:52:01.201608 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:52:01.201501 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnzft\" (UniqueName: \"kubernetes.io/projected/8282fcb6-7612-4ade-9386-3b1e7ae11b13-kube-api-access-pnzft\") pod \"maas-api-e2e-ipp-b-b37ba5-d668d8449-xqgrq\" (UID: \"8282fcb6-7612-4ade-9386-3b1e7ae11b13\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b37ba5-d668d8449-xqgrq" Nov 03 01:52:01.202013 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:52:01.201987 5437 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-b37ba5-serving-cert: secret "maas-api-e2e-ipp-b-b37ba5-serving-cert" not found Nov 03 01:52:01.202221 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:52:01.202205 5437 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8282fcb6-7612-4ade-9386-3b1e7ae11b13-maas-api-tls podName:8282fcb6-7612-4ade-9386-3b1e7ae11b13 nodeName:}" failed. No retries permitted until 2026-11-03 01:52:01.702182243 +0000 UTC m=+2003.731664518 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8282fcb6-7612-4ade-9386-3b1e7ae11b13-maas-api-tls") pod "maas-api-e2e-ipp-b-b37ba5-d668d8449-xqgrq" (UID: "8282fcb6-7612-4ade-9386-3b1e7ae11b13") : secret "maas-api-e2e-ipp-b-b37ba5-serving-cert" not found Nov 03 01:52:01.202381 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:52:01.202200 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8282fcb6-7612-4ade-9386-3b1e7ae11b13-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-b37ba5-d668d8449-xqgrq\" (UID: \"8282fcb6-7612-4ade-9386-3b1e7ae11b13\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b37ba5-d668d8449-xqgrq" Nov 03 01:52:01.204550 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:52:01.204529 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8282fcb6-7612-4ade-9386-3b1e7ae11b13-metrics-certs\") pod \"maas-api-e2e-ipp-b-b37ba5-d668d8449-xqgrq\" (UID: \"8282fcb6-7612-4ade-9386-3b1e7ae11b13\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b37ba5-d668d8449-xqgrq" Nov 03 01:52:01.265248 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:52:01.265213 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnzft\" (UniqueName: \"kubernetes.io/projected/8282fcb6-7612-4ade-9386-3b1e7ae11b13-kube-api-access-pnzft\") pod \"maas-api-e2e-ipp-b-b37ba5-d668d8449-xqgrq\" (UID: \"8282fcb6-7612-4ade-9386-3b1e7ae11b13\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b37ba5-d668d8449-xqgrq" Nov 03 01:52:01.321826 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:52:01.321731 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:52:01.525460 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:52:01.525421 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:52:01.707820 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:52:01.707723 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8282fcb6-7612-4ade-9386-3b1e7ae11b13-maas-api-tls\") pod \"maas-api-e2e-ipp-b-b37ba5-d668d8449-xqgrq\" (UID: \"8282fcb6-7612-4ade-9386-3b1e7ae11b13\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b37ba5-d668d8449-xqgrq" Nov 03 01:52:01.710424 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:52:01.710355 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8282fcb6-7612-4ade-9386-3b1e7ae11b13-maas-api-tls\") pod \"maas-api-e2e-ipp-b-b37ba5-d668d8449-xqgrq\" (UID: \"8282fcb6-7612-4ade-9386-3b1e7ae11b13\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b37ba5-d668d8449-xqgrq" Nov 03 01:52:01.730518 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:52:01.730479 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:52:01.981174 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:52:01.981087 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b37ba5-d668d8449-xqgrq" Nov 03 01:52:02.359274 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:52:02.358618 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8282fcb6_7612_4ade_9386_3b1e7ae11b13.slice/crio-9d76bc86132e0ee825496dcc320b8ca5da20acfcaf5c7af5bff0cd2d7611851b WatchSource:0}: Error finding container 9d76bc86132e0ee825496dcc320b8ca5da20acfcaf5c7af5bff0cd2d7611851b: Status 404 returned error can't find the container with id 9d76bc86132e0ee825496dcc320b8ca5da20acfcaf5c7af5bff0cd2d7611851b Nov 03 01:52:02.360999 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:52:02.360969 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b37ba5-d668d8449-xqgrq"] Nov 03 01:52:02.819186 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:52:02.819151 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b37ba5-d668d8449-xqgrq" event={"ID":"8282fcb6-7612-4ade-9386-3b1e7ae11b13","Type":"ContainerStarted","Data":"9d76bc86132e0ee825496dcc320b8ca5da20acfcaf5c7af5bff0cd2d7611851b"} Nov 03 01:52:03.830023 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:52:03.829982 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b37ba5-d668d8449-xqgrq" event={"ID":"8282fcb6-7612-4ade-9386-3b1e7ae11b13","Type":"ContainerStarted","Data":"1602df86bb1babdd2570e8b02e9b5f30354020a19bff31b4e0cb67221fd25f85"} Nov 03 01:52:03.830515 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:52:03.830129 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b37ba5-d668d8449-xqgrq" Nov 03 01:52:03.952726 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:52:03.952671 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b37ba5-d668d8449-xqgrq" podStartSLOduration=2.059116776 podStartE2EDuration="2.952655525s" totalImagesPullingTime="893.538749ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:52:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:52:03.951099538 +0000 UTC m=+2005.980581834" watchObservedRunningTime="2026-11-03 01:52:03.952655525 +0000 UTC m=+2005.982137819" Nov 03 01:52:04.041900 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:52:04.041860 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:52:05.037870 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:52:05.037824 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:52:05.530495 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:52:05.530459 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-738acb-openshift-default-c7bc4fcf7-llk59"] Nov 03 01:52:05.539925 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:52:05.539889 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-738acb-openshift-default-c7bc4fcf7-llk59" Nov 03 01:52:05.593871 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:52:05.593834 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-738acb-openshift-default-c7bc4fcf7-llk59"] Nov 03 01:52:05.641119 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:52:05.641077 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-738acb-openshift-default-dockercfg-9xnvw\"" Nov 03 01:52:05.655788 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:52:05.655756 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:52:05.657018 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:52:05.656989 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a06e089e-fba5-403f-9fe5-34175c6865fa-istio-data\") pod \"e2e-ipp-b-738acb-openshift-default-c7bc4fcf7-llk59\" (UID: \"a06e089e-fba5-403f-9fe5-34175c6865fa\") " pod="openshift-ingress/e2e-ipp-b-738acb-openshift-default-c7bc4fcf7-llk59" Nov 03 01:52:05.657163 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:52:05.657029 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7l5x\" (UniqueName: \"kubernetes.io/projected/a06e089e-fba5-403f-9fe5-34175c6865fa-kube-api-access-t7l5x\") pod \"e2e-ipp-b-738acb-openshift-default-c7bc4fcf7-llk59\" (UID: \"a06e089e-fba5-403f-9fe5-34175c6865fa\") " pod="openshift-ingress/e2e-ipp-b-738acb-openshift-default-c7bc4fcf7-llk59" Nov 03 01:52:05.657228 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:52:05.657157 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a06e089e-fba5-403f-9fe5-34175c6865fa-istio-token\") pod \"e2e-ipp-b-738acb-openshift-default-c7bc4fcf7-llk59\" (UID: \"a06e089e-fba5-403f-9fe5-34175c6865fa\") " pod="openshift-ingress/e2e-ipp-b-738acb-openshift-default-c7bc4fcf7-llk59" Nov 03 01:52:05.657228 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:52:05.657197 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a06e089e-fba5-403f-9fe5-34175c6865fa-istiod-ca-cert\") pod \"e2e-ipp-b-738acb-openshift-default-c7bc4fcf7-llk59\" (UID: \"a06e089e-fba5-403f-9fe5-34175c6865fa\") " pod="openshift-ingress/e2e-ipp-b-738acb-openshift-default-c7bc4fcf7-llk59" Nov 03 01:52:05.657228 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:52:05.657217 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a06e089e-fba5-403f-9fe5-34175c6865fa-workload-certs\") pod \"e2e-ipp-b-738acb-openshift-default-c7bc4fcf7-llk59\" (UID: \"a06e089e-fba5-403f-9fe5-34175c6865fa\") " pod="openshift-ingress/e2e-ipp-b-738acb-openshift-default-c7bc4fcf7-llk59" Nov 03 01:52:05.657380 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:52:05.657247 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a06e089e-fba5-403f-9fe5-34175c6865fa-workload-socket\") pod \"e2e-ipp-b-738acb-openshift-default-c7bc4fcf7-llk59\" (UID: \"a06e089e-fba5-403f-9fe5-34175c6865fa\") " pod="openshift-ingress/e2e-ipp-b-738acb-openshift-default-c7bc4fcf7-llk59" Nov 03 01:52:05.657380 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:52:05.657306 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a06e089e-fba5-403f-9fe5-34175c6865fa-istio-envoy\") pod \"e2e-ipp-b-738acb-openshift-default-c7bc4fcf7-llk59\" (UID: \"a06e089e-fba5-403f-9fe5-34175c6865fa\") " pod="openshift-ingress/e2e-ipp-b-738acb-openshift-default-c7bc4fcf7-llk59" Nov 03 01:52:05.657487 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:52:05.657472 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a06e089e-fba5-403f-9fe5-34175c6865fa-istio-podinfo\") pod \"e2e-ipp-b-738acb-openshift-default-c7bc4fcf7-llk59\" (UID: \"a06e089e-fba5-403f-9fe5-34175c6865fa\") " pod="openshift-ingress/e2e-ipp-b-738acb-openshift-default-c7bc4fcf7-llk59" Nov 03 01:52:05.657533 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:52:05.657504 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a06e089e-fba5-403f-9fe5-34175c6865fa-credential-socket\") pod \"e2e-ipp-b-738acb-openshift-default-c7bc4fcf7-llk59\" (UID: \"a06e089e-fba5-403f-9fe5-34175c6865fa\") " pod="openshift-ingress/e2e-ipp-b-738acb-openshift-default-c7bc4fcf7-llk59" Nov 03 01:52:05.759256 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:52:05.759189 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a06e089e-fba5-403f-9fe5-34175c6865fa-istio-podinfo\") pod \"e2e-ipp-b-738acb-openshift-default-c7bc4fcf7-llk59\" (UID: \"a06e089e-fba5-403f-9fe5-34175c6865fa\") " pod="openshift-ingress/e2e-ipp-b-738acb-openshift-default-c7bc4fcf7-llk59" Nov 03 01:52:05.760396 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:52:05.760348 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a06e089e-fba5-403f-9fe5-34175c6865fa-credential-socket\") pod \"e2e-ipp-b-738acb-openshift-default-c7bc4fcf7-llk59\" (UID: \"a06e089e-fba5-403f-9fe5-34175c6865fa\") " pod="openshift-ingress/e2e-ipp-b-738acb-openshift-default-c7bc4fcf7-llk59" Nov 03 01:52:05.760700 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:52:05.760679 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a06e089e-fba5-403f-9fe5-34175c6865fa-istio-data\") pod \"e2e-ipp-b-738acb-openshift-default-c7bc4fcf7-llk59\" (UID: \"a06e089e-fba5-403f-9fe5-34175c6865fa\") " pod="openshift-ingress/e2e-ipp-b-738acb-openshift-default-c7bc4fcf7-llk59" Nov 03 01:52:05.760868 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:52:05.760850 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7l5x\" (UniqueName: \"kubernetes.io/projected/a06e089e-fba5-403f-9fe5-34175c6865fa-kube-api-access-t7l5x\") pod \"e2e-ipp-b-738acb-openshift-default-c7bc4fcf7-llk59\" (UID: \"a06e089e-fba5-403f-9fe5-34175c6865fa\") " pod="openshift-ingress/e2e-ipp-b-738acb-openshift-default-c7bc4fcf7-llk59" Nov 03 01:52:05.761107 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:52:05.761087 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a06e089e-fba5-403f-9fe5-34175c6865fa-istio-token\") pod \"e2e-ipp-b-738acb-openshift-default-c7bc4fcf7-llk59\" (UID: \"a06e089e-fba5-403f-9fe5-34175c6865fa\") " pod="openshift-ingress/e2e-ipp-b-738acb-openshift-default-c7bc4fcf7-llk59" Nov 03 01:52:05.761582 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:52:05.761560 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a06e089e-fba5-403f-9fe5-34175c6865fa-istiod-ca-cert\") pod \"e2e-ipp-b-738acb-openshift-default-c7bc4fcf7-llk59\" (UID: \"a06e089e-fba5-403f-9fe5-34175c6865fa\") " pod="openshift-ingress/e2e-ipp-b-738acb-openshift-default-c7bc4fcf7-llk59" Nov 03 01:52:05.761715 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:52:05.761697 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a06e089e-fba5-403f-9fe5-34175c6865fa-workload-certs\") pod \"e2e-ipp-b-738acb-openshift-default-c7bc4fcf7-llk59\" (UID: \"a06e089e-fba5-403f-9fe5-34175c6865fa\") " pod="openshift-ingress/e2e-ipp-b-738acb-openshift-default-c7bc4fcf7-llk59" Nov 03 01:52:05.761843 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:52:05.761828 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a06e089e-fba5-403f-9fe5-34175c6865fa-workload-socket\") pod \"e2e-ipp-b-738acb-openshift-default-c7bc4fcf7-llk59\" (UID: \"a06e089e-fba5-403f-9fe5-34175c6865fa\") " pod="openshift-ingress/e2e-ipp-b-738acb-openshift-default-c7bc4fcf7-llk59" Nov 03 01:52:05.761952 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:52:05.761935 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a06e089e-fba5-403f-9fe5-34175c6865fa-istio-envoy\") pod \"e2e-ipp-b-738acb-openshift-default-c7bc4fcf7-llk59\" (UID: \"a06e089e-fba5-403f-9fe5-34175c6865fa\") " pod="openshift-ingress/e2e-ipp-b-738acb-openshift-default-c7bc4fcf7-llk59" Nov 03 01:52:05.762942 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:52:05.762907 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a06e089e-fba5-403f-9fe5-34175c6865fa-istio-data\") pod \"e2e-ipp-b-738acb-openshift-default-c7bc4fcf7-llk59\" (UID: \"a06e089e-fba5-403f-9fe5-34175c6865fa\") " pod="openshift-ingress/e2e-ipp-b-738acb-openshift-default-c7bc4fcf7-llk59" Nov 03 01:52:05.763046 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:52:05.763025 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a06e089e-fba5-403f-9fe5-34175c6865fa-credential-socket\") pod \"e2e-ipp-b-738acb-openshift-default-c7bc4fcf7-llk59\" (UID: \"a06e089e-fba5-403f-9fe5-34175c6865fa\") " pod="openshift-ingress/e2e-ipp-b-738acb-openshift-default-c7bc4fcf7-llk59" Nov 03 01:52:05.763714 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:52:05.763689 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a06e089e-fba5-403f-9fe5-34175c6865fa-workload-socket\") pod \"e2e-ipp-b-738acb-openshift-default-c7bc4fcf7-llk59\" (UID: \"a06e089e-fba5-403f-9fe5-34175c6865fa\") " pod="openshift-ingress/e2e-ipp-b-738acb-openshift-default-c7bc4fcf7-llk59" Nov 03 01:52:05.764626 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:52:05.764599 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a06e089e-fba5-403f-9fe5-34175c6865fa-istiod-ca-cert\") pod \"e2e-ipp-b-738acb-openshift-default-c7bc4fcf7-llk59\" (UID: \"a06e089e-fba5-403f-9fe5-34175c6865fa\") " pod="openshift-ingress/e2e-ipp-b-738acb-openshift-default-c7bc4fcf7-llk59" Nov 03 01:52:05.767423 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:52:05.767061 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a06e089e-fba5-403f-9fe5-34175c6865fa-workload-certs\") pod \"e2e-ipp-b-738acb-openshift-default-c7bc4fcf7-llk59\" (UID: \"a06e089e-fba5-403f-9fe5-34175c6865fa\") " pod="openshift-ingress/e2e-ipp-b-738acb-openshift-default-c7bc4fcf7-llk59" Nov 03 01:52:05.767690 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:52:05.767665 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a06e089e-fba5-403f-9fe5-34175c6865fa-istio-envoy\") pod \"e2e-ipp-b-738acb-openshift-default-c7bc4fcf7-llk59\" (UID: \"a06e089e-fba5-403f-9fe5-34175c6865fa\") " pod="openshift-ingress/e2e-ipp-b-738acb-openshift-default-c7bc4fcf7-llk59" Nov 03 01:52:05.784672 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:52:05.784584 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a06e089e-fba5-403f-9fe5-34175c6865fa-istio-podinfo\") pod \"e2e-ipp-b-738acb-openshift-default-c7bc4fcf7-llk59\" (UID: \"a06e089e-fba5-403f-9fe5-34175c6865fa\") " pod="openshift-ingress/e2e-ipp-b-738acb-openshift-default-c7bc4fcf7-llk59" Nov 03 01:52:05.836258 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:52:05.836222 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a06e089e-fba5-403f-9fe5-34175c6865fa-istio-token\") pod \"e2e-ipp-b-738acb-openshift-default-c7bc4fcf7-llk59\" (UID: \"a06e089e-fba5-403f-9fe5-34175c6865fa\") " pod="openshift-ingress/e2e-ipp-b-738acb-openshift-default-c7bc4fcf7-llk59" Nov 03 01:52:05.836546 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:52:05.836523 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7l5x\" (UniqueName: \"kubernetes.io/projected/a06e089e-fba5-403f-9fe5-34175c6865fa-kube-api-access-t7l5x\") pod \"e2e-ipp-b-738acb-openshift-default-c7bc4fcf7-llk59\" (UID: \"a06e089e-fba5-403f-9fe5-34175c6865fa\") " pod="openshift-ingress/e2e-ipp-b-738acb-openshift-default-c7bc4fcf7-llk59" Nov 03 01:52:05.858739 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:52:05.858704 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:52:05.864182 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:52:05.864153 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-738acb-openshift-default-c7bc4fcf7-llk59" Nov 03 01:52:06.064064 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:52:06.064022 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:52:06.310875 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:52:06.310817 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda06e089e_fba5_403f_9fe5_34175c6865fa.slice/crio-c46faf9323563e066085344b2efdc5a74c122cb470433015f354672ad16e6038 WatchSource:0}: Error finding container c46faf9323563e066085344b2efdc5a74c122cb470433015f354672ad16e6038: Status 404 returned error can't find the container with id c46faf9323563e066085344b2efdc5a74c122cb470433015f354672ad16e6038 Nov 03 01:52:06.313941 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:52:06.313908 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 01:52:06.314025 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:52:06.313977 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 01:52:06.314025 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:52:06.313980 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-738acb-openshift-default-c7bc4fcf7-llk59"] Nov 03 01:52:06.314025 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:52:06.314010 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 01:52:06.401762 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:52:06.401720 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:52:06.853917 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:52:06.853878 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-738acb-openshift-default-c7bc4fcf7-llk59" event={"ID":"a06e089e-fba5-403f-9fe5-34175c6865fa","Type":"ContainerStarted","Data":"af249d8e97dad4103572480f15fd4ce304a5912a0d4aee49f3017d7960f2ecee"} Nov 03 01:52:06.854113 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:52:06.853940 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-738acb-openshift-default-c7bc4fcf7-llk59" event={"ID":"a06e089e-fba5-403f-9fe5-34175c6865fa","Type":"ContainerStarted","Data":"c46faf9323563e066085344b2efdc5a74c122cb470433015f354672ad16e6038"} Nov 03 01:52:06.865231 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:52:06.865205 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-738acb-openshift-default-c7bc4fcf7-llk59" Nov 03 01:52:06.977879 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:52:06.977830 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-738acb-openshift-default-c7bc4fcf7-llk59" podStartSLOduration=1.9778155819999998 podStartE2EDuration="1.977815582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:52:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:52:06.976866181 +0000 UTC m=+2009.006348489" watchObservedRunningTime="2026-11-03 01:52:06.977815582 +0000 UTC m=+2009.007297878" Nov 03 01:52:07.871787 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:52:07.871746 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-738acb-openshift-default-c7bc4fcf7-llk59" Nov 03 01:52:08.866745 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:52:08.866715 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-738acb-openshift-default-c7bc4fcf7-llk59" Nov 03 01:52:08.867752 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:52:08.867729 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-738acb-openshift-default-c7bc4fcf7-llk59" Nov 03 01:52:09.842690 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:52:09.842660 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b37ba5-d668d8449-xqgrq" Nov 03 01:52:10.559535 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:52:10.559494 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:52:10.820776 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:52:10.820636 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:52:11.642345 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:52:11.642298 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:52:13.780147 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:52:13.780097 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:52:13.982664 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:52:13.982612 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:52:14.036358 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:52:14.036269 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 01:52:18.509685 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:52:18.509645 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:52:23.473517 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:52:23.473474 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:52:23.812942 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:52:23.812905 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:52:24.071933 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:52:24.071835 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:52:28.352447 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:52:28.352403 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:52:33.620097 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:52:33.620055 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:52:33.821979 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:52:33.821941 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:52:34.672983 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:52:34.672942 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:52:39.347225 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:52:39.347185 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:52:39.555348 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:52:39.555304 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:52:39.759299 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:52:39.759200 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:52:44.037423 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:52:44.037380 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 01:52:44.290884 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:52:44.290763 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:52:47.328188 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:52:47.328139 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:52:49.086117 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:52:49.086070 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:52:53.990997 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:52:53.990955 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:52:57.648743 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:52:57.648693 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:52:58.114848 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:52:58.114808 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:52:59.917617 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:52:59.917573 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:53:00.200673 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:00.200586 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-71746a-86854c59ff-x7lbf"] Nov 03 01:53:00.309436 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:00.309401 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-71746a-6f445fbc6c-sbmzg"] Nov 03 01:53:00.614402 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:00.614329 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:53:00.861960 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:00.861925 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:53:01.521205 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:01.521170 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-hj9f6"] Nov 03 01:53:01.521662 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:01.521401 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-hj9f6" podUID="6b6b29be-71cc-442f-b430-0cc2344d400d" containerName="istio-proxy" containerID="cri-o://9ae126f2a6a19e771de01ad82e4bfff14a7ed70738a87af663320230aea7117e" gracePeriod=30 Nov 03 01:53:02.112695 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:02.112659 5437 patch_prober.go:29] interesting pod/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-hj9f6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 01:53:02.112868 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:02.112748 5437 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-hj9f6" podUID="6b6b29be-71cc-442f-b430-0cc2344d400d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 01:53:05.201327 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:05.201263 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-71746a-86854c59ff-x7lbf" podUID="d6f96330-e093-4c72-9501-83baed0edee8" containerName="payload-processing" containerID="cri-o://a4c83a294afd5791edde30dd8ab6bd43e0302342582d6c19fea6549d3173c62d" gracePeriod=25 Nov 03 01:53:05.310436 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:05.310385 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-71746a-6f445fbc6c-sbmzg" podUID="4c0b0e96-c609-4127-ab55-c0c8f11e4e01" containerName="payload-pre-processing" containerID="cri-o://9381d0f3c8c30c6f18de476dc08c44c75ad83e976608f606cdc413240e22da4d" gracePeriod=25 Nov 03 01:53:05.481731 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:05.481695 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-71746a-86854c59ff-x7lbf" Nov 03 01:53:05.578181 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:05.578148 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6f96330-e093-4c72-9501-83baed0edee8-kube-api-access-dmqgq\" (UniqueName: \"kubernetes.io/projected/d6f96330-e093-4c72-9501-83baed0edee8-kube-api-access-dmqgq\") pod \"d6f96330-e093-4c72-9501-83baed0edee8\" (UID: \"d6f96330-e093-4c72-9501-83baed0edee8\") " Nov 03 01:53:05.578344 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:05.578300 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6f96330-e093-4c72-9501-83baed0edee8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6f96330-e093-4c72-9501-83baed0edee8-plugins-config-volume\") pod \"d6f96330-e093-4c72-9501-83baed0edee8\" (UID: \"d6f96330-e093-4c72-9501-83baed0edee8\") " Nov 03 01:53:05.578679 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:05.578651 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6f96330-e093-4c72-9501-83baed0edee8-plugins-config-volume" pod "d6f96330-e093-4c72-9501-83baed0edee8" (UID: "d6f96330-e093-4c72-9501-83baed0edee8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:53:05.580329 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:05.580304 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6f96330-e093-4c72-9501-83baed0edee8-kube-api-access-dmqgq" pod "d6f96330-e093-4c72-9501-83baed0edee8" (UID: "d6f96330-e093-4c72-9501-83baed0edee8"). InnerVolumeSpecName "kube-api-access-dmqgq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:53:05.593870 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:05.593838 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-71746a-6f445fbc6c-sbmzg" Nov 03 01:53:05.651423 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:05.651360 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:53:05.680133 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:05.680101 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c0b0e96-c609-4127-ab55-c0c8f11e4e01-kube-api-access-76znl\" (UniqueName: \"kubernetes.io/projected/4c0b0e96-c609-4127-ab55-c0c8f11e4e01-kube-api-access-76znl\") pod \"4c0b0e96-c609-4127-ab55-c0c8f11e4e01\" (UID: \"4c0b0e96-c609-4127-ab55-c0c8f11e4e01\") " Nov 03 01:53:05.680321 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:05.680266 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c0b0e96-c609-4127-ab55-c0c8f11e4e01-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c0b0e96-c609-4127-ab55-c0c8f11e4e01-plugins-config-volume\") pod \"4c0b0e96-c609-4127-ab55-c0c8f11e4e01\" (UID: \"4c0b0e96-c609-4127-ab55-c0c8f11e4e01\") " Nov 03 01:53:05.680736 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:05.680707 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c0b0e96-c609-4127-ab55-c0c8f11e4e01-plugins-config-volume" pod "4c0b0e96-c609-4127-ab55-c0c8f11e4e01" (UID: "4c0b0e96-c609-4127-ab55-c0c8f11e4e01"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:53:05.680845 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:05.680772 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmqgq\" (UniqueName: \"kubernetes.io/projected/d6f96330-e093-4c72-9501-83baed0edee8-kube-api-access-dmqgq\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:53:05.680845 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:05.680789 5437 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6f96330-e093-4c72-9501-83baed0edee8-plugins-config-volume\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:53:05.682348 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:05.682322 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c0b0e96-c609-4127-ab55-c0c8f11e4e01-kube-api-access-76znl" pod "4c0b0e96-c609-4127-ab55-c0c8f11e4e01" (UID: "4c0b0e96-c609-4127-ab55-c0c8f11e4e01"). InnerVolumeSpecName "kube-api-access-76znl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:53:05.782542 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:05.782449 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76znl\" (UniqueName: \"kubernetes.io/projected/4c0b0e96-c609-4127-ab55-c0c8f11e4e01-kube-api-access-76znl\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:53:05.782542 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:05.782480 5437 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c0b0e96-c609-4127-ab55-c0c8f11e4e01-plugins-config-volume\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:53:06.243068 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:06.243026 5437 generic.go:356] "Generic (PLEG): container finished" podID="d6f96330-e093-4c72-9501-83baed0edee8" containerID="a4c83a294afd5791edde30dd8ab6bd43e0302342582d6c19fea6549d3173c62d" exitCode=0 Nov 03 01:53:06.243521 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:06.243097 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-71746a-86854c59ff-x7lbf" Nov 03 01:53:06.243521 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:06.243114 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-71746a-86854c59ff-x7lbf" event={"ID":"d6f96330-e093-4c72-9501-83baed0edee8","Type":"ContainerDied","Data":"a4c83a294afd5791edde30dd8ab6bd43e0302342582d6c19fea6549d3173c62d"} Nov 03 01:53:06.243521 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:06.243172 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-71746a-86854c59ff-x7lbf" event={"ID":"d6f96330-e093-4c72-9501-83baed0edee8","Type":"ContainerDied","Data":"91288973d584aa2be9a49b6b15a12c3b6b41fa5e372cc3da9ddafe762b3e4305"} Nov 03 01:53:06.243521 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:06.243222 5437 scope.go:122] "RemoveContainer" containerID="a4c83a294afd5791edde30dd8ab6bd43e0302342582d6c19fea6549d3173c62d" Nov 03 01:53:06.245255 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:06.245223 5437 generic.go:356] "Generic (PLEG): container finished" podID="4c0b0e96-c609-4127-ab55-c0c8f11e4e01" containerID="9381d0f3c8c30c6f18de476dc08c44c75ad83e976608f606cdc413240e22da4d" exitCode=0 Nov 03 01:53:06.245412 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:06.245286 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-71746a-6f445fbc6c-sbmzg" Nov 03 01:53:06.245492 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:06.245420 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-71746a-6f445fbc6c-sbmzg" event={"ID":"4c0b0e96-c609-4127-ab55-c0c8f11e4e01","Type":"ContainerDied","Data":"9381d0f3c8c30c6f18de476dc08c44c75ad83e976608f606cdc413240e22da4d"} Nov 03 01:53:06.245492 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:06.245460 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-71746a-6f445fbc6c-sbmzg" event={"ID":"4c0b0e96-c609-4127-ab55-c0c8f11e4e01","Type":"ContainerDied","Data":"56a3b4cd59852f73d5eaf30a4456ef84636d1daa403dd60a03bd245a7276361f"} Nov 03 01:53:06.256084 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:06.256053 5437 scope.go:122] "RemoveContainer" containerID="a4c83a294afd5791edde30dd8ab6bd43e0302342582d6c19fea6549d3173c62d" Nov 03 01:53:06.256526 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:53:06.256500 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4c83a294afd5791edde30dd8ab6bd43e0302342582d6c19fea6549d3173c62d\": container with ID starting with a4c83a294afd5791edde30dd8ab6bd43e0302342582d6c19fea6549d3173c62d not found: ID does not exist" containerID="a4c83a294afd5791edde30dd8ab6bd43e0302342582d6c19fea6549d3173c62d" Nov 03 01:53:06.256639 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:06.256534 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4c83a294afd5791edde30dd8ab6bd43e0302342582d6c19fea6549d3173c62d"} err="failed to get container status \"a4c83a294afd5791edde30dd8ab6bd43e0302342582d6c19fea6549d3173c62d\": rpc error: code = NotFound desc = could not find container \"a4c83a294afd5791edde30dd8ab6bd43e0302342582d6c19fea6549d3173c62d\": container with ID starting with a4c83a294afd5791edde30dd8ab6bd43e0302342582d6c19fea6549d3173c62d not found: ID does not exist" Nov 03 01:53:06.256639 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:06.256555 5437 scope.go:122] "RemoveContainer" containerID="9381d0f3c8c30c6f18de476dc08c44c75ad83e976608f606cdc413240e22da4d" Nov 03 01:53:06.271463 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:06.271435 5437 scope.go:122] "RemoveContainer" containerID="9381d0f3c8c30c6f18de476dc08c44c75ad83e976608f606cdc413240e22da4d" Nov 03 01:53:06.271893 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:53:06.271870 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9381d0f3c8c30c6f18de476dc08c44c75ad83e976608f606cdc413240e22da4d\": container with ID starting with 9381d0f3c8c30c6f18de476dc08c44c75ad83e976608f606cdc413240e22da4d not found: ID does not exist" containerID="9381d0f3c8c30c6f18de476dc08c44c75ad83e976608f606cdc413240e22da4d" Nov 03 01:53:06.271990 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:06.271905 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9381d0f3c8c30c6f18de476dc08c44c75ad83e976608f606cdc413240e22da4d"} err="failed to get container status \"9381d0f3c8c30c6f18de476dc08c44c75ad83e976608f606cdc413240e22da4d\": rpc error: code = NotFound desc = could not find container \"9381d0f3c8c30c6f18de476dc08c44c75ad83e976608f606cdc413240e22da4d\": container with ID starting with 9381d0f3c8c30c6f18de476dc08c44c75ad83e976608f606cdc413240e22da4d not found: ID does not exist" Nov 03 01:53:06.456110 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:06.456020 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-71746a-6f445fbc6c-sbmzg"] Nov 03 01:53:06.460149 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:06.460105 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-71746a-6f445fbc6c-sbmzg"] Nov 03 01:53:06.577767 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:06.577714 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-71746a-86854c59ff-x7lbf"] Nov 03 01:53:06.581023 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:06.580972 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-71746a-86854c59ff-x7lbf"] Nov 03 01:53:06.796999 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:06.796966 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-hj9f6" Nov 03 01:53:06.892442 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:06.892386 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6b6b29be-71cc-442f-b430-0cc2344d400d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b6b29be-71cc-442f-b430-0cc2344d400d-istio-podinfo\") pod \"6b6b29be-71cc-442f-b430-0cc2344d400d\" (UID: \"6b6b29be-71cc-442f-b430-0cc2344d400d\") " Nov 03 01:53:06.892648 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:06.892484 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b6b29be-71cc-442f-b430-0cc2344d400d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b6b29be-71cc-442f-b430-0cc2344d400d-workload-socket\") pod \"6b6b29be-71cc-442f-b430-0cc2344d400d\" (UID: \"6b6b29be-71cc-442f-b430-0cc2344d400d\") " Nov 03 01:53:06.892648 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:06.892606 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b6b29be-71cc-442f-b430-0cc2344d400d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b6b29be-71cc-442f-b430-0cc2344d400d-credential-socket\") pod \"6b6b29be-71cc-442f-b430-0cc2344d400d\" (UID: \"6b6b29be-71cc-442f-b430-0cc2344d400d\") " Nov 03 01:53:06.892746 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:06.892683 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b6b29be-71cc-442f-b430-0cc2344d400d-kube-api-access-vqwrq\" (UniqueName: \"kubernetes.io/projected/6b6b29be-71cc-442f-b430-0cc2344d400d-kube-api-access-vqwrq\") pod \"6b6b29be-71cc-442f-b430-0cc2344d400d\" (UID: \"6b6b29be-71cc-442f-b430-0cc2344d400d\") " Nov 03 01:53:06.892746 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:06.892715 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b6b29be-71cc-442f-b430-0cc2344d400d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b6b29be-71cc-442f-b430-0cc2344d400d-istio-data\") pod \"6b6b29be-71cc-442f-b430-0cc2344d400d\" (UID: \"6b6b29be-71cc-442f-b430-0cc2344d400d\") " Nov 03 01:53:06.892838 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:06.892768 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b6b29be-71cc-442f-b430-0cc2344d400d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b6b29be-71cc-442f-b430-0cc2344d400d-istiod-ca-cert\") pod \"6b6b29be-71cc-442f-b430-0cc2344d400d\" (UID: \"6b6b29be-71cc-442f-b430-0cc2344d400d\") " Nov 03 01:53:06.892838 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:06.892821 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b6b29be-71cc-442f-b430-0cc2344d400d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b6b29be-71cc-442f-b430-0cc2344d400d-istio-envoy\") pod \"6b6b29be-71cc-442f-b430-0cc2344d400d\" (UID: \"6b6b29be-71cc-442f-b430-0cc2344d400d\") " Nov 03 01:53:06.892838 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:06.892819 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b6b29be-71cc-442f-b430-0cc2344d400d-workload-socket" pod "6b6b29be-71cc-442f-b430-0cc2344d400d" (UID: "6b6b29be-71cc-442f-b430-0cc2344d400d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:53:06.892999 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:06.892856 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b6b29be-71cc-442f-b430-0cc2344d400d-istio-token\" (UniqueName: \"kubernetes.io/projected/6b6b29be-71cc-442f-b430-0cc2344d400d-istio-token\") pod \"6b6b29be-71cc-442f-b430-0cc2344d400d\" (UID: \"6b6b29be-71cc-442f-b430-0cc2344d400d\") " Nov 03 01:53:06.892999 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:06.892873 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b6b29be-71cc-442f-b430-0cc2344d400d-credential-socket" pod "6b6b29be-71cc-442f-b430-0cc2344d400d" (UID: "6b6b29be-71cc-442f-b430-0cc2344d400d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:53:06.892999 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:06.892896 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b6b29be-71cc-442f-b430-0cc2344d400d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b6b29be-71cc-442f-b430-0cc2344d400d-workload-certs\") pod \"6b6b29be-71cc-442f-b430-0cc2344d400d\" (UID: \"6b6b29be-71cc-442f-b430-0cc2344d400d\") " Nov 03 01:53:06.893157 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:06.893068 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b6b29be-71cc-442f-b430-0cc2344d400d-istio-data" pod "6b6b29be-71cc-442f-b430-0cc2344d400d" (UID: "6b6b29be-71cc-442f-b430-0cc2344d400d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:53:06.893322 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:06.893281 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b6b29be-71cc-442f-b430-0cc2344d400d-workload-certs" pod "6b6b29be-71cc-442f-b430-0cc2344d400d" (UID: "6b6b29be-71cc-442f-b430-0cc2344d400d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:53:06.893508 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:06.893402 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b6b29be-71cc-442f-b430-0cc2344d400d-istiod-ca-cert" pod "6b6b29be-71cc-442f-b430-0cc2344d400d" (UID: "6b6b29be-71cc-442f-b430-0cc2344d400d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:53:06.893612 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:06.893587 5437 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b6b29be-71cc-442f-b430-0cc2344d400d-credential-socket\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:53:06.893717 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:06.893616 5437 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b6b29be-71cc-442f-b430-0cc2344d400d-istio-data\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:53:06.893717 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:06.893633 5437 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b6b29be-71cc-442f-b430-0cc2344d400d-istiod-ca-cert\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:53:06.893717 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:06.893648 5437 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b6b29be-71cc-442f-b430-0cc2344d400d-workload-certs\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:53:06.893717 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:06.893662 5437 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b6b29be-71cc-442f-b430-0cc2344d400d-workload-socket\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:53:06.894910 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:06.894886 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6b6b29be-71cc-442f-b430-0cc2344d400d-istio-podinfo" pod "6b6b29be-71cc-442f-b430-0cc2344d400d" (UID: "6b6b29be-71cc-442f-b430-0cc2344d400d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:53:06.895003 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:06.894974 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b6b29be-71cc-442f-b430-0cc2344d400d-kube-api-access-vqwrq" pod "6b6b29be-71cc-442f-b430-0cc2344d400d" (UID: "6b6b29be-71cc-442f-b430-0cc2344d400d"). InnerVolumeSpecName "kube-api-access-vqwrq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:53:06.895003 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:06.894992 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b6b29be-71cc-442f-b430-0cc2344d400d-istio-envoy" pod "6b6b29be-71cc-442f-b430-0cc2344d400d" (UID: "6b6b29be-71cc-442f-b430-0cc2344d400d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:53:06.895127 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:06.895111 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b6b29be-71cc-442f-b430-0cc2344d400d-istio-token" pod "6b6b29be-71cc-442f-b430-0cc2344d400d" (UID: "6b6b29be-71cc-442f-b430-0cc2344d400d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:53:06.994689 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:06.994646 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqwrq\" (UniqueName: \"kubernetes.io/projected/6b6b29be-71cc-442f-b430-0cc2344d400d-kube-api-access-vqwrq\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:53:06.994689 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:06.994684 5437 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b6b29be-71cc-442f-b430-0cc2344d400d-istio-envoy\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:53:06.994689 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:06.994693 5437 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b6b29be-71cc-442f-b430-0cc2344d400d-istio-token\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:53:06.994949 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:06.994702 5437 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b6b29be-71cc-442f-b430-0cc2344d400d-istio-podinfo\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:53:07.254952 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:07.254911 5437 generic.go:356] "Generic (PLEG): container finished" podID="6b6b29be-71cc-442f-b430-0cc2344d400d" containerID="9ae126f2a6a19e771de01ad82e4bfff14a7ed70738a87af663320230aea7117e" exitCode=0 Nov 03 01:53:07.255473 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:07.254976 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-hj9f6" Nov 03 01:53:07.255473 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:07.254997 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-hj9f6" event={"ID":"6b6b29be-71cc-442f-b430-0cc2344d400d","Type":"ContainerDied","Data":"9ae126f2a6a19e771de01ad82e4bfff14a7ed70738a87af663320230aea7117e"} Nov 03 01:53:07.255473 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:07.255067 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-hj9f6" event={"ID":"6b6b29be-71cc-442f-b430-0cc2344d400d","Type":"ContainerDied","Data":"58878f70fc646fd4a8b70417b5e938fbc42a47f565f5eb016fec5486c792d1bf"} Nov 03 01:53:07.255473 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:07.255098 5437 scope.go:122] "RemoveContainer" containerID="9ae126f2a6a19e771de01ad82e4bfff14a7ed70738a87af663320230aea7117e" Nov 03 01:53:07.266487 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:07.266463 5437 scope.go:122] "RemoveContainer" containerID="9ae126f2a6a19e771de01ad82e4bfff14a7ed70738a87af663320230aea7117e" Nov 03 01:53:07.266824 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:53:07.266798 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ae126f2a6a19e771de01ad82e4bfff14a7ed70738a87af663320230aea7117e\": container with ID starting with 9ae126f2a6a19e771de01ad82e4bfff14a7ed70738a87af663320230aea7117e not found: ID does not exist" containerID="9ae126f2a6a19e771de01ad82e4bfff14a7ed70738a87af663320230aea7117e" Nov 03 01:53:07.266930 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:07.266835 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ae126f2a6a19e771de01ad82e4bfff14a7ed70738a87af663320230aea7117e"} err="failed to get container status \"9ae126f2a6a19e771de01ad82e4bfff14a7ed70738a87af663320230aea7117e\": rpc error: code = NotFound desc = could not find container \"9ae126f2a6a19e771de01ad82e4bfff14a7ed70738a87af663320230aea7117e\": container with ID starting with 9ae126f2a6a19e771de01ad82e4bfff14a7ed70738a87af663320230aea7117e not found: ID does not exist" Nov 03 01:53:07.425934 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:07.425899 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-hj9f6"] Nov 03 01:53:07.477622 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:07.477584 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-hj9f6"] Nov 03 01:53:08.090200 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:08.090164 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c0b0e96-c609-4127-ab55-c0c8f11e4e01" path="/var/lib/kubelet/pods/4c0b0e96-c609-4127-ab55-c0c8f11e4e01/volumes" Nov 03 01:53:08.090727 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:08.090712 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b6b29be-71cc-442f-b430-0cc2344d400d" path="/var/lib/kubelet/pods/6b6b29be-71cc-442f-b430-0cc2344d400d/volumes" Nov 03 01:53:08.091073 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:08.091060 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6f96330-e093-4c72-9501-83baed0edee8" path="/var/lib/kubelet/pods/d6f96330-e093-4c72-9501-83baed0edee8/volumes" Nov 03 01:53:11.999407 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:11.999317 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:53:14.034776 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:53:14.034737 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 01:53:17.034180 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:17.034084 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:53:19.926134 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:19.926086 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:53:20.454034 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:20.453900 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:53:20.917987 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:20.917895 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:53:23.585776 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:23.585734 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:53:25.774014 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:25.773958 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:53:28.537463 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:28.537416 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:53:29.996522 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:29.996483 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b37ba5-847f4b6f4c-lt2jx"] Nov 03 01:53:30.049137 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:30.049101 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:53:30.116258 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:30.115994 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b37ba5-75c4b6866c-tnvxd"] Nov 03 01:53:30.445903 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:30.445762 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:53:30.672093 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:30.672053 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:53:33.202179 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:33.202144 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-738acb-6c99bd8c8f-8zf4p"] Nov 03 01:53:33.202812 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:33.202787 5437 cpu_manager.go:414] "containerMap: removing container" podUID="6b6b29be-71cc-442f-b430-0cc2344d400d" containerName="istio-proxy" Nov 03 01:53:33.202812 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:33.202808 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b6b29be-71cc-442f-b430-0cc2344d400d" containerName="istio-proxy" Nov 03 01:53:33.202966 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:33.202827 5437 cpu_manager.go:414] "containerMap: removing container" podUID="d6f96330-e093-4c72-9501-83baed0edee8" containerName="payload-processing" Nov 03 01:53:33.202966 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:33.202832 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6f96330-e093-4c72-9501-83baed0edee8" containerName="payload-processing" Nov 03 01:53:33.202966 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:33.202839 5437 cpu_manager.go:414] "containerMap: removing container" podUID="4c0b0e96-c609-4127-ab55-c0c8f11e4e01" containerName="payload-pre-processing" Nov 03 01:53:33.202966 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:33.202844 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c0b0e96-c609-4127-ab55-c0c8f11e4e01" containerName="payload-pre-processing" Nov 03 01:53:33.202966 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:33.202951 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6f96330-e093-4c72-9501-83baed0edee8" containerName="payload-processing" Nov 03 01:53:33.202966 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:33.202961 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c0b0e96-c609-4127-ab55-c0c8f11e4e01" containerName="payload-pre-processing" Nov 03 01:53:33.203152 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:33.202973 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b6b29be-71cc-442f-b430-0cc2344d400d" containerName="istio-proxy" Nov 03 01:53:33.208902 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:33.208860 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-738acb-6c99bd8c8f-8zf4p" Nov 03 01:53:33.261835 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:33.261795 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-738acb-6c99bd8c8f-8zf4p"] Nov 03 01:53:33.309955 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:33.309916 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-738acb-serving-cert\"" Nov 03 01:53:33.342740 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:33.342697 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/306b59aa-822c-4e6c-9434-bc06393c2802-maas-api-tls\") pod \"maas-api-e2e-ipp-b-738acb-6c99bd8c8f-8zf4p\" (UID: \"306b59aa-822c-4e6c-9434-bc06393c2802\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-738acb-6c99bd8c8f-8zf4p" Nov 03 01:53:33.343064 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:33.343031 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vvxs\" (UniqueName: \"kubernetes.io/projected/306b59aa-822c-4e6c-9434-bc06393c2802-kube-api-access-9vvxs\") pod \"maas-api-e2e-ipp-b-738acb-6c99bd8c8f-8zf4p\" (UID: \"306b59aa-822c-4e6c-9434-bc06393c2802\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-738acb-6c99bd8c8f-8zf4p" Nov 03 01:53:33.343223 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:33.343207 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/306b59aa-822c-4e6c-9434-bc06393c2802-metrics-certs\") pod \"maas-api-e2e-ipp-b-738acb-6c99bd8c8f-8zf4p\" (UID: \"306b59aa-822c-4e6c-9434-bc06393c2802\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-738acb-6c99bd8c8f-8zf4p" Nov 03 01:53:33.343354 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:33.343257 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/306b59aa-822c-4e6c-9434-bc06393c2802-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-738acb-6c99bd8c8f-8zf4p\" (UID: \"306b59aa-822c-4e6c-9434-bc06393c2802\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-738acb-6c99bd8c8f-8zf4p" Nov 03 01:53:33.444688 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:33.444643 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/306b59aa-822c-4e6c-9434-bc06393c2802-maas-api-tls\") pod \"maas-api-e2e-ipp-b-738acb-6c99bd8c8f-8zf4p\" (UID: \"306b59aa-822c-4e6c-9434-bc06393c2802\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-738acb-6c99bd8c8f-8zf4p" Nov 03 01:53:33.444886 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:33.444703 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vvxs\" (UniqueName: \"kubernetes.io/projected/306b59aa-822c-4e6c-9434-bc06393c2802-kube-api-access-9vvxs\") pod \"maas-api-e2e-ipp-b-738acb-6c99bd8c8f-8zf4p\" (UID: \"306b59aa-822c-4e6c-9434-bc06393c2802\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-738acb-6c99bd8c8f-8zf4p" Nov 03 01:53:33.444886 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:33.444801 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/306b59aa-822c-4e6c-9434-bc06393c2802-metrics-certs\") pod \"maas-api-e2e-ipp-b-738acb-6c99bd8c8f-8zf4p\" (UID: \"306b59aa-822c-4e6c-9434-bc06393c2802\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-738acb-6c99bd8c8f-8zf4p" Nov 03 01:53:33.444886 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:33.444854 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/306b59aa-822c-4e6c-9434-bc06393c2802-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-738acb-6c99bd8c8f-8zf4p\" (UID: \"306b59aa-822c-4e6c-9434-bc06393c2802\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-738acb-6c99bd8c8f-8zf4p" Nov 03 01:53:33.445613 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:33.445587 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/306b59aa-822c-4e6c-9434-bc06393c2802-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-738acb-6c99bd8c8f-8zf4p\" (UID: \"306b59aa-822c-4e6c-9434-bc06393c2802\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-738acb-6c99bd8c8f-8zf4p" Nov 03 01:53:33.447275 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:33.447248 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/306b59aa-822c-4e6c-9434-bc06393c2802-metrics-certs\") pod \"maas-api-e2e-ipp-b-738acb-6c99bd8c8f-8zf4p\" (UID: \"306b59aa-822c-4e6c-9434-bc06393c2802\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-738acb-6c99bd8c8f-8zf4p" Nov 03 01:53:33.447425 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:33.447356 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/306b59aa-822c-4e6c-9434-bc06393c2802-maas-api-tls\") pod \"maas-api-e2e-ipp-b-738acb-6c99bd8c8f-8zf4p\" (UID: \"306b59aa-822c-4e6c-9434-bc06393c2802\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-738acb-6c99bd8c8f-8zf4p" Nov 03 01:53:33.461906 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:33.461830 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:53:33.507856 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:33.507820 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vvxs\" (UniqueName: \"kubernetes.io/projected/306b59aa-822c-4e6c-9434-bc06393c2802-kube-api-access-9vvxs\") pod \"maas-api-e2e-ipp-b-738acb-6c99bd8c8f-8zf4p\" (UID: \"306b59aa-822c-4e6c-9434-bc06393c2802\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-738acb-6c99bd8c8f-8zf4p" Nov 03 01:53:33.525627 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:33.525575 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-738acb-6c99bd8c8f-8zf4p" Nov 03 01:53:33.669762 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:33.669716 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:53:33.903274 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:53:33.903240 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod306b59aa_822c_4e6c_9434_bc06393c2802.slice/crio-3fe06aab1a69c97b37de3e30e604253cc892b1c87a6fa3b95e8efa219c596283 WatchSource:0}: Error finding container 3fe06aab1a69c97b37de3e30e604253cc892b1c87a6fa3b95e8efa219c596283: Status 404 returned error can't find the container with id 3fe06aab1a69c97b37de3e30e604253cc892b1c87a6fa3b95e8efa219c596283 Nov 03 01:53:33.905839 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:33.905814 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-738acb-6c99bd8c8f-8zf4p"] Nov 03 01:53:34.487982 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:34.487943 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-738acb-6c99bd8c8f-8zf4p" event={"ID":"306b59aa-822c-4e6c-9434-bc06393c2802","Type":"ContainerStarted","Data":"3fe06aab1a69c97b37de3e30e604253cc892b1c87a6fa3b95e8efa219c596283"} Nov 03 01:53:34.997682 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:34.997635 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-b37ba5-847f4b6f4c-lt2jx" podUID="04cd0e8c-c175-4c88-a5d3-856e5c04fe98" containerName="payload-processing" containerID="cri-o://8d0600e8a97ab1a07d92795008375cfc72d2f7981aade7de51624ecb25d3bfff" gracePeriod=25 Nov 03 01:53:35.120888 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:35.120852 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b37ba5-75c4b6866c-tnvxd" podUID="d37572bf-d59f-4e9a-8745-f4d91be336f5" containerName="payload-pre-processing" containerID="cri-o://7f25928b0e73e9f487e73bf4a29769c6ef1c88b7487356095c66e143d3e60688" gracePeriod=25 Nov 03 01:53:35.286339 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:35.286305 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b37ba5-847f4b6f4c-lt2jx" Nov 03 01:53:35.371806 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:35.371688 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04cd0e8c-c175-4c88-a5d3-856e5c04fe98-kube-api-access-lt657\" (UniqueName: \"kubernetes.io/projected/04cd0e8c-c175-4c88-a5d3-856e5c04fe98-kube-api-access-lt657\") pod \"04cd0e8c-c175-4c88-a5d3-856e5c04fe98\" (UID: \"04cd0e8c-c175-4c88-a5d3-856e5c04fe98\") " Nov 03 01:53:35.371806 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:35.371749 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04cd0e8c-c175-4c88-a5d3-856e5c04fe98-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04cd0e8c-c175-4c88-a5d3-856e5c04fe98-plugins-config-volume\") pod \"04cd0e8c-c175-4c88-a5d3-856e5c04fe98\" (UID: \"04cd0e8c-c175-4c88-a5d3-856e5c04fe98\") " Nov 03 01:53:35.373184 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:35.372241 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04cd0e8c-c175-4c88-a5d3-856e5c04fe98-plugins-config-volume" pod "04cd0e8c-c175-4c88-a5d3-856e5c04fe98" (UID: "04cd0e8c-c175-4c88-a5d3-856e5c04fe98"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:53:35.373184 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:35.372798 5437 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04cd0e8c-c175-4c88-a5d3-856e5c04fe98-plugins-config-volume\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:53:35.377848 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:35.377818 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04cd0e8c-c175-4c88-a5d3-856e5c04fe98-kube-api-access-lt657" pod "04cd0e8c-c175-4c88-a5d3-856e5c04fe98" (UID: "04cd0e8c-c175-4c88-a5d3-856e5c04fe98"). InnerVolumeSpecName "kube-api-access-lt657". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:53:35.387092 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:35.387058 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b37ba5-75c4b6866c-tnvxd" Nov 03 01:53:35.473826 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:35.473789 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d37572bf-d59f-4e9a-8745-f4d91be336f5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d37572bf-d59f-4e9a-8745-f4d91be336f5-plugins-config-volume\") pod \"d37572bf-d59f-4e9a-8745-f4d91be336f5\" (UID: \"d37572bf-d59f-4e9a-8745-f4d91be336f5\") " Nov 03 01:53:35.474012 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:35.473925 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d37572bf-d59f-4e9a-8745-f4d91be336f5-kube-api-access-7z2q9\" (UniqueName: \"kubernetes.io/projected/d37572bf-d59f-4e9a-8745-f4d91be336f5-kube-api-access-7z2q9\") pod \"d37572bf-d59f-4e9a-8745-f4d91be336f5\" (UID: \"d37572bf-d59f-4e9a-8745-f4d91be336f5\") " Nov 03 01:53:35.474216 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:35.474189 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d37572bf-d59f-4e9a-8745-f4d91be336f5-plugins-config-volume" pod "d37572bf-d59f-4e9a-8745-f4d91be336f5" (UID: "d37572bf-d59f-4e9a-8745-f4d91be336f5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:53:35.474484 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:35.474468 5437 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d37572bf-d59f-4e9a-8745-f4d91be336f5-plugins-config-volume\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:53:35.474547 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:35.474487 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lt657\" (UniqueName: \"kubernetes.io/projected/04cd0e8c-c175-4c88-a5d3-856e5c04fe98-kube-api-access-lt657\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:53:35.476183 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:35.476156 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d37572bf-d59f-4e9a-8745-f4d91be336f5-kube-api-access-7z2q9" pod "d37572bf-d59f-4e9a-8745-f4d91be336f5" (UID: "d37572bf-d59f-4e9a-8745-f4d91be336f5"). InnerVolumeSpecName "kube-api-access-7z2q9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:53:35.496021 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:35.495982 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-738acb-6c99bd8c8f-8zf4p" event={"ID":"306b59aa-822c-4e6c-9434-bc06393c2802","Type":"ContainerStarted","Data":"46c0256b12beeb4f44be6462fe25c3e4c6cd2761bf893e938eed6430ccf95d1f"} Nov 03 01:53:35.496535 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:35.496050 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-738acb-6c99bd8c8f-8zf4p" Nov 03 01:53:35.497524 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:35.497499 5437 generic.go:356] "Generic (PLEG): container finished" podID="d37572bf-d59f-4e9a-8745-f4d91be336f5" containerID="7f25928b0e73e9f487e73bf4a29769c6ef1c88b7487356095c66e143d3e60688" exitCode=0 Nov 03 01:53:35.497647 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:35.497539 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b37ba5-75c4b6866c-tnvxd" Nov 03 01:53:35.497647 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:35.497574 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b37ba5-75c4b6866c-tnvxd" event={"ID":"d37572bf-d59f-4e9a-8745-f4d91be336f5","Type":"ContainerDied","Data":"7f25928b0e73e9f487e73bf4a29769c6ef1c88b7487356095c66e143d3e60688"} Nov 03 01:53:35.497647 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:35.497618 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b37ba5-75c4b6866c-tnvxd" event={"ID":"d37572bf-d59f-4e9a-8745-f4d91be336f5","Type":"ContainerDied","Data":"c6b22fac60e24fee665f36eb7cdf0e38e07d54016f277214498f1eb3ba7fec42"} Nov 03 01:53:35.497647 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:35.497640 5437 scope.go:122] "RemoveContainer" containerID="7f25928b0e73e9f487e73bf4a29769c6ef1c88b7487356095c66e143d3e60688" Nov 03 01:53:35.499149 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:35.499126 5437 generic.go:356] "Generic (PLEG): container finished" podID="04cd0e8c-c175-4c88-a5d3-856e5c04fe98" containerID="8d0600e8a97ab1a07d92795008375cfc72d2f7981aade7de51624ecb25d3bfff" exitCode=0 Nov 03 01:53:35.499270 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:35.499179 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b37ba5-847f4b6f4c-lt2jx" Nov 03 01:53:35.499270 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:35.499192 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b37ba5-847f4b6f4c-lt2jx" event={"ID":"04cd0e8c-c175-4c88-a5d3-856e5c04fe98","Type":"ContainerDied","Data":"8d0600e8a97ab1a07d92795008375cfc72d2f7981aade7de51624ecb25d3bfff"} Nov 03 01:53:35.499270 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:35.499230 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b37ba5-847f4b6f4c-lt2jx" event={"ID":"04cd0e8c-c175-4c88-a5d3-856e5c04fe98","Type":"ContainerDied","Data":"4cb22e6cb5c3541e15ed18301bb93ac98d6f69f8a84e31f4d1b018bf9b775883"} Nov 03 01:53:35.509939 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:35.509913 5437 scope.go:122] "RemoveContainer" containerID="7f25928b0e73e9f487e73bf4a29769c6ef1c88b7487356095c66e143d3e60688" Nov 03 01:53:35.510333 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:53:35.510307 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f25928b0e73e9f487e73bf4a29769c6ef1c88b7487356095c66e143d3e60688\": container with ID starting with 7f25928b0e73e9f487e73bf4a29769c6ef1c88b7487356095c66e143d3e60688 not found: ID does not exist" containerID="7f25928b0e73e9f487e73bf4a29769c6ef1c88b7487356095c66e143d3e60688" Nov 03 01:53:35.510441 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:35.510342 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f25928b0e73e9f487e73bf4a29769c6ef1c88b7487356095c66e143d3e60688"} err="failed to get container status \"7f25928b0e73e9f487e73bf4a29769c6ef1c88b7487356095c66e143d3e60688\": rpc error: code = NotFound desc = could not find container \"7f25928b0e73e9f487e73bf4a29769c6ef1c88b7487356095c66e143d3e60688\": container with ID starting with 7f25928b0e73e9f487e73bf4a29769c6ef1c88b7487356095c66e143d3e60688 not found: ID does not exist" Nov 03 01:53:35.510441 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:35.510394 5437 scope.go:122] "RemoveContainer" containerID="8d0600e8a97ab1a07d92795008375cfc72d2f7981aade7de51624ecb25d3bfff" Nov 03 01:53:35.520584 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:35.520563 5437 scope.go:122] "RemoveContainer" containerID="8d0600e8a97ab1a07d92795008375cfc72d2f7981aade7de51624ecb25d3bfff" Nov 03 01:53:35.520919 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:53:35.520898 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d0600e8a97ab1a07d92795008375cfc72d2f7981aade7de51624ecb25d3bfff\": container with ID starting with 8d0600e8a97ab1a07d92795008375cfc72d2f7981aade7de51624ecb25d3bfff not found: ID does not exist" containerID="8d0600e8a97ab1a07d92795008375cfc72d2f7981aade7de51624ecb25d3bfff" Nov 03 01:53:35.520967 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:35.520931 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d0600e8a97ab1a07d92795008375cfc72d2f7981aade7de51624ecb25d3bfff"} err="failed to get container status \"8d0600e8a97ab1a07d92795008375cfc72d2f7981aade7de51624ecb25d3bfff\": rpc error: code = NotFound desc = could not find container \"8d0600e8a97ab1a07d92795008375cfc72d2f7981aade7de51624ecb25d3bfff\": container with ID starting with 8d0600e8a97ab1a07d92795008375cfc72d2f7981aade7de51624ecb25d3bfff not found: ID does not exist" Nov 03 01:53:35.575945 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:35.575893 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7z2q9\" (UniqueName: \"kubernetes.io/projected/d37572bf-d59f-4e9a-8745-f4d91be336f5-kube-api-access-7z2q9\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:53:35.653910 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:35.653847 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-738acb-6c99bd8c8f-8zf4p" podStartSLOduration=1.297027959 podStartE2EDuration="2.653831082s" totalImagesPullingTime="1.356803123s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:53:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:53:35.651163552 +0000 UTC m=+2097.680645847" watchObservedRunningTime="2026-11-03 01:53:35.653831082 +0000 UTC m=+2097.683313376" Nov 03 01:53:35.773247 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:35.772896 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b37ba5-75c4b6866c-tnvxd"] Nov 03 01:53:35.778109 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:35.778070 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b37ba5-75c4b6866c-tnvxd"] Nov 03 01:53:35.883334 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:35.883295 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b37ba5-847f4b6f4c-lt2jx"] Nov 03 01:53:35.886511 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:35.886486 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b37ba5-847f4b6f4c-lt2jx"] Nov 03 01:53:36.089175 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:36.089130 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04cd0e8c-c175-4c88-a5d3-856e5c04fe98" path="/var/lib/kubelet/pods/04cd0e8c-c175-4c88-a5d3-856e5c04fe98/volumes" Nov 03 01:53:36.089690 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:36.089668 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d37572bf-d59f-4e9a-8745-f4d91be336f5" path="/var/lib/kubelet/pods/d37572bf-d59f-4e9a-8745-f4d91be336f5/volumes" Nov 03 01:53:38.975745 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:38.975641 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:53:41.519572 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:41.519539 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-738acb-6c99bd8c8f-8zf4p" Nov 03 01:53:44.035690 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:53:44.035651 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 01:53:45.289224 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:45.289179 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:53:47.655927 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:47.655888 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:53:50.105079 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:50.105034 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:53:54.252597 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:54.252548 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:53:54.745718 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:54.745673 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:53:54.952342 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:54.952299 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:53:56.171965 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:56.171918 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:53:58.145267 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:58.145228 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:53:58.357009 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:58.356971 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:53:59.763387 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:53:59.763172 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:54:00.377240 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:54:00.377197 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:54:01.223004 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:54:01.222958 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:54:01.474643 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:54:01.474559 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:54:04.700561 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:54:04.700525 5437 generic.go:356] "Generic (PLEG): container finished" podID="a06e089e-fba5-403f-9fe5-34175c6865fa" containerID="af249d8e97dad4103572480f15fd4ce304a5912a0d4aee49f3017d7960f2ecee" exitCode=0 Nov 03 01:54:04.701014 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:54:04.700597 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-738acb-openshift-default-c7bc4fcf7-llk59" event={"ID":"a06e089e-fba5-403f-9fe5-34175c6865fa","Type":"ContainerDied","Data":"af249d8e97dad4103572480f15fd4ce304a5912a0d4aee49f3017d7960f2ecee"} Nov 03 01:54:04.701060 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:54:04.701018 5437 scope.go:122] "RemoveContainer" containerID="af249d8e97dad4103572480f15fd4ce304a5912a0d4aee49f3017d7960f2ecee" Nov 03 01:54:04.702652 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:54:04.702619 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 01:54:04.702782 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:54:04.702710 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 01:54:04.702782 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:54:04.702760 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 01:54:05.721456 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:54:05.721336 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-738acb-openshift-default-c7bc4fcf7-llk59" event={"ID":"a06e089e-fba5-403f-9fe5-34175c6865fa","Type":"ContainerStarted","Data":"fdded550c5957428c7f6d533d35877beeabf2bd33a185dc03a2ae3c1c1d821b2"} Nov 03 01:54:05.864826 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:54:05.864788 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-738acb-openshift-default-c7bc4fcf7-llk59" Nov 03 01:54:05.866156 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:54:05.866127 5437 patch_prober.go:29] interesting pod/e2e-ipp-b-738acb-openshift-default-c7bc4fcf7-llk59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Nov 03 01:54:05.866310 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:54:05.866186 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-738acb-openshift-default-c7bc4fcf7-llk59" podUID="a06e089e-fba5-403f-9fe5-34175c6865fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Nov 03 01:54:06.549987 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:54:06.549933 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:54:06.865143 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:54:06.865049 5437 patch_prober.go:29] interesting pod/e2e-ipp-b-738acb-openshift-default-c7bc4fcf7-llk59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Nov 03 01:54:06.865143 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:54:06.865109 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-738acb-openshift-default-c7bc4fcf7-llk59" podUID="a06e089e-fba5-403f-9fe5-34175c6865fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Nov 03 01:54:07.344629 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:54:07.344590 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:54:07.864951 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:54:07.864912 5437 patch_prober.go:29] interesting pod/e2e-ipp-b-738acb-openshift-default-c7bc4fcf7-llk59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Nov 03 01:54:07.865152 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:54:07.864996 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-738acb-openshift-default-c7bc4fcf7-llk59" podUID="a06e089e-fba5-403f-9fe5-34175c6865fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Nov 03 01:54:08.864643 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:54:08.864603 5437 patch_prober.go:29] interesting pod/e2e-ipp-b-738acb-openshift-default-c7bc4fcf7-llk59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Nov 03 01:54:08.865151 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:54:08.864671 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-738acb-openshift-default-c7bc4fcf7-llk59" podUID="a06e089e-fba5-403f-9fe5-34175c6865fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Nov 03 01:54:08.865846 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:54:08.865818 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:54:08.866618 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:54:08.866595 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-738acb-openshift-default-c7bc4fcf7-llk59" Nov 03 01:54:09.244257 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:54:09.244168 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-2w7h6"] Nov 03 01:54:09.244464 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:54:09.244434 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-2w7h6" podUID="1aa3545f-2866-4899-8e50-1b55db8a4b1b" containerName="istio-proxy" containerID="cri-o://cdc63d9b0d92517e5191cddc28deafcd772da29fbbe48ca9d3db9c7c916099f1" gracePeriod=30 Nov 03 01:54:09.864676 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:54:09.864636 5437 patch_prober.go:29] interesting pod/e2e-ipp-b-738acb-openshift-default-c7bc4fcf7-llk59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Nov 03 01:54:09.865233 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:54:09.864708 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-738acb-openshift-default-c7bc4fcf7-llk59" podUID="a06e089e-fba5-403f-9fe5-34175c6865fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Nov 03 01:54:10.543910 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:54:10.543866 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:54:10.865157 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:54:10.865117 5437 patch_prober.go:29] interesting pod/e2e-ipp-b-738acb-openshift-default-c7bc4fcf7-llk59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Nov 03 01:54:10.865571 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:54:10.865176 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-738acb-openshift-default-c7bc4fcf7-llk59" podUID="a06e089e-fba5-403f-9fe5-34175c6865fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Nov 03 01:54:11.870273 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:54:11.870234 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-738acb-openshift-default-c7bc4fcf7-llk59" Nov 03 01:54:11.871297 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:54:11.871275 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-738acb-openshift-default-c7bc4fcf7-llk59" Nov 03 01:54:12.010497 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:54:12.010452 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:54:13.835146 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:54:13.835103 5437 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-2w7h6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 01:54:13.835602 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:54:13.835194 5437 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-2w7h6" podUID="1aa3545f-2866-4899-8e50-1b55db8a4b1b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 01:54:14.035149 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:54:14.035110 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 01:54:14.518335 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:54:14.518278 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-2w7h6" Nov 03 01:54:14.634156 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:54:14.634120 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1aa3545f-2866-4899-8e50-1b55db8a4b1b-kube-api-access-46n8n\" (UniqueName: \"kubernetes.io/projected/1aa3545f-2866-4899-8e50-1b55db8a4b1b-kube-api-access-46n8n\") pod \"1aa3545f-2866-4899-8e50-1b55db8a4b1b\" (UID: \"1aa3545f-2866-4899-8e50-1b55db8a4b1b\") " Nov 03 01:54:14.634336 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:54:14.634201 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1aa3545f-2866-4899-8e50-1b55db8a4b1b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1aa3545f-2866-4899-8e50-1b55db8a4b1b-istio-envoy\") pod \"1aa3545f-2866-4899-8e50-1b55db8a4b1b\" (UID: \"1aa3545f-2866-4899-8e50-1b55db8a4b1b\") " Nov 03 01:54:14.634421 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:54:14.634384 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1aa3545f-2866-4899-8e50-1b55db8a4b1b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1aa3545f-2866-4899-8e50-1b55db8a4b1b-istio-podinfo\") pod \"1aa3545f-2866-4899-8e50-1b55db8a4b1b\" (UID: \"1aa3545f-2866-4899-8e50-1b55db8a4b1b\") " Nov 03 01:54:14.634520 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:54:14.634502 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1aa3545f-2866-4899-8e50-1b55db8a4b1b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1aa3545f-2866-4899-8e50-1b55db8a4b1b-workload-certs\") pod \"1aa3545f-2866-4899-8e50-1b55db8a4b1b\" (UID: \"1aa3545f-2866-4899-8e50-1b55db8a4b1b\") " Nov 03 01:54:14.634628 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:54:14.634610 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1aa3545f-2866-4899-8e50-1b55db8a4b1b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1aa3545f-2866-4899-8e50-1b55db8a4b1b-istiod-ca-cert\") pod \"1aa3545f-2866-4899-8e50-1b55db8a4b1b\" (UID: \"1aa3545f-2866-4899-8e50-1b55db8a4b1b\") " Nov 03 01:54:14.634707 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:54:14.634686 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1aa3545f-2866-4899-8e50-1b55db8a4b1b-istio-token\" (UniqueName: \"kubernetes.io/projected/1aa3545f-2866-4899-8e50-1b55db8a4b1b-istio-token\") pod \"1aa3545f-2866-4899-8e50-1b55db8a4b1b\" (UID: \"1aa3545f-2866-4899-8e50-1b55db8a4b1b\") " Nov 03 01:54:14.634825 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:54:14.634785 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1aa3545f-2866-4899-8e50-1b55db8a4b1b-workload-certs" pod "1aa3545f-2866-4899-8e50-1b55db8a4b1b" (UID: "1aa3545f-2866-4899-8e50-1b55db8a4b1b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:54:14.634825 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:54:14.634809 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1aa3545f-2866-4899-8e50-1b55db8a4b1b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1aa3545f-2866-4899-8e50-1b55db8a4b1b-istio-data\") pod \"1aa3545f-2866-4899-8e50-1b55db8a4b1b\" (UID: \"1aa3545f-2866-4899-8e50-1b55db8a4b1b\") " Nov 03 01:54:14.634966 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:54:14.634850 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1aa3545f-2866-4899-8e50-1b55db8a4b1b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aa3545f-2866-4899-8e50-1b55db8a4b1b-credential-socket\") pod \"1aa3545f-2866-4899-8e50-1b55db8a4b1b\" (UID: \"1aa3545f-2866-4899-8e50-1b55db8a4b1b\") " Nov 03 01:54:14.634966 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:54:14.634900 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1aa3545f-2866-4899-8e50-1b55db8a4b1b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aa3545f-2866-4899-8e50-1b55db8a4b1b-workload-socket\") pod \"1aa3545f-2866-4899-8e50-1b55db8a4b1b\" (UID: \"1aa3545f-2866-4899-8e50-1b55db8a4b1b\") " Nov 03 01:54:14.635078 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:54:14.635020 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1aa3545f-2866-4899-8e50-1b55db8a4b1b-istio-data" pod "1aa3545f-2866-4899-8e50-1b55db8a4b1b" (UID: "1aa3545f-2866-4899-8e50-1b55db8a4b1b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:54:14.635078 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:54:14.635043 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1aa3545f-2866-4899-8e50-1b55db8a4b1b-istiod-ca-cert" pod "1aa3545f-2866-4899-8e50-1b55db8a4b1b" (UID: "1aa3545f-2866-4899-8e50-1b55db8a4b1b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:54:14.635230 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:54:14.635209 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1aa3545f-2866-4899-8e50-1b55db8a4b1b-credential-socket" pod "1aa3545f-2866-4899-8e50-1b55db8a4b1b" (UID: "1aa3545f-2866-4899-8e50-1b55db8a4b1b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:54:14.635473 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:54:14.635446 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1aa3545f-2866-4899-8e50-1b55db8a4b1b-workload-socket" pod "1aa3545f-2866-4899-8e50-1b55db8a4b1b" (UID: "1aa3545f-2866-4899-8e50-1b55db8a4b1b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:54:14.635571 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:54:14.635513 5437 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1aa3545f-2866-4899-8e50-1b55db8a4b1b-istio-data\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:54:14.635571 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:54:14.635537 5437 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aa3545f-2866-4899-8e50-1b55db8a4b1b-credential-socket\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:54:14.635571 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:54:14.635551 5437 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1aa3545f-2866-4899-8e50-1b55db8a4b1b-workload-certs\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:54:14.635571 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:54:14.635565 5437 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1aa3545f-2866-4899-8e50-1b55db8a4b1b-istiod-ca-cert\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:54:14.636529 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:54:14.636503 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1aa3545f-2866-4899-8e50-1b55db8a4b1b-kube-api-access-46n8n" pod "1aa3545f-2866-4899-8e50-1b55db8a4b1b" (UID: "1aa3545f-2866-4899-8e50-1b55db8a4b1b"). InnerVolumeSpecName "kube-api-access-46n8n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:54:14.636630 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:54:14.636575 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1aa3545f-2866-4899-8e50-1b55db8a4b1b-istio-podinfo" pod "1aa3545f-2866-4899-8e50-1b55db8a4b1b" (UID: "1aa3545f-2866-4899-8e50-1b55db8a4b1b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:54:14.636749 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:54:14.636735 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1aa3545f-2866-4899-8e50-1b55db8a4b1b-istio-envoy" pod "1aa3545f-2866-4899-8e50-1b55db8a4b1b" (UID: "1aa3545f-2866-4899-8e50-1b55db8a4b1b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:54:14.636925 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:54:14.636908 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1aa3545f-2866-4899-8e50-1b55db8a4b1b-istio-token" pod "1aa3545f-2866-4899-8e50-1b55db8a4b1b" (UID: "1aa3545f-2866-4899-8e50-1b55db8a4b1b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:54:14.736819 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:54:14.736782 5437 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1aa3545f-2866-4899-8e50-1b55db8a4b1b-istio-token\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:54:14.736819 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:54:14.736816 5437 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aa3545f-2866-4899-8e50-1b55db8a4b1b-workload-socket\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:54:14.737036 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:54:14.736828 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46n8n\" (UniqueName: \"kubernetes.io/projected/1aa3545f-2866-4899-8e50-1b55db8a4b1b-kube-api-access-46n8n\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:54:14.737036 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:54:14.736839 5437 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1aa3545f-2866-4899-8e50-1b55db8a4b1b-istio-envoy\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:54:14.737036 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:54:14.736847 5437 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1aa3545f-2866-4899-8e50-1b55db8a4b1b-istio-podinfo\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:54:14.803318 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:54:14.803269 5437 generic.go:356] "Generic (PLEG): container finished" podID="1aa3545f-2866-4899-8e50-1b55db8a4b1b" containerID="cdc63d9b0d92517e5191cddc28deafcd772da29fbbe48ca9d3db9c7c916099f1" exitCode=0 Nov 03 01:54:14.803527 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:54:14.803323 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-2w7h6" Nov 03 01:54:14.803527 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:54:14.803348 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-2w7h6" event={"ID":"1aa3545f-2866-4899-8e50-1b55db8a4b1b","Type":"ContainerDied","Data":"cdc63d9b0d92517e5191cddc28deafcd772da29fbbe48ca9d3db9c7c916099f1"} Nov 03 01:54:14.803527 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:54:14.803441 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-2w7h6" event={"ID":"1aa3545f-2866-4899-8e50-1b55db8a4b1b","Type":"ContainerDied","Data":"b2e02594bfa12e4497dcc44f61266d015342aa4e7448ddfae82215cf38de9c5a"} Nov 03 01:54:14.803527 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:54:14.803464 5437 scope.go:122] "RemoveContainer" containerID="cdc63d9b0d92517e5191cddc28deafcd772da29fbbe48ca9d3db9c7c916099f1" Nov 03 01:54:14.814088 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:54:14.814066 5437 scope.go:122] "RemoveContainer" containerID="cdc63d9b0d92517e5191cddc28deafcd772da29fbbe48ca9d3db9c7c916099f1" Nov 03 01:54:14.814326 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:54:14.814308 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdc63d9b0d92517e5191cddc28deafcd772da29fbbe48ca9d3db9c7c916099f1\": container with ID starting with cdc63d9b0d92517e5191cddc28deafcd772da29fbbe48ca9d3db9c7c916099f1 not found: ID does not exist" containerID="cdc63d9b0d92517e5191cddc28deafcd772da29fbbe48ca9d3db9c7c916099f1" Nov 03 01:54:14.814397 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:54:14.814335 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdc63d9b0d92517e5191cddc28deafcd772da29fbbe48ca9d3db9c7c916099f1"} err="failed to get container status \"cdc63d9b0d92517e5191cddc28deafcd772da29fbbe48ca9d3db9c7c916099f1\": rpc error: code = NotFound desc = could not find container \"cdc63d9b0d92517e5191cddc28deafcd772da29fbbe48ca9d3db9c7c916099f1\": container with ID starting with cdc63d9b0d92517e5191cddc28deafcd772da29fbbe48ca9d3db9c7c916099f1 not found: ID does not exist" Nov 03 01:54:14.985148 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:54:14.985111 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-2w7h6"] Nov 03 01:54:14.989278 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:54:14.989247 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-2w7h6"] Nov 03 01:54:15.649092 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:54:15.649042 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:54:16.091480 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:54:16.091445 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1aa3545f-2866-4899-8e50-1b55db8a4b1b" path="/var/lib/kubelet/pods/1aa3545f-2866-4899-8e50-1b55db8a4b1b/volumes" Nov 03 01:54:16.200428 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:54:16.200354 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:54:17.033313 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:54:17.033278 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:54:17.523522 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:54:17.523492 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:54:17.723214 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:54:17.723174 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:54:18.378317 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:54:18.378282 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:54:18.860891 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:54:18.860849 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:54:19.138245 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:54:19.138163 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:54:20.230570 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:54:20.230529 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:54:21.863786 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:54:21.863735 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:54:22.121061 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:54:22.120969 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:54:23.124773 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:54:23.124737 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:54:23.360097 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:54:23.360060 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:54:23.860399 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:54:23.860321 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:54:25.476533 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:54:25.476495 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:54:25.684426 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:54:25.684389 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:54:26.080893 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:54:26.080841 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:54:26.288525 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:54:26.288488 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:54:30.297768 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:54:30.297708 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:54:30.506342 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:54:30.506306 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:54:32.363660 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:54:32.363624 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:54:33.426281 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:54:33.426235 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:54:33.631635 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:54:33.631595 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:54:35.417582 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:54:35.417548 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:54:37.386106 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:54:37.386009 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:54:41.345321 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:54:41.345274 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:54:44.036146 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:54:44.036104 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 01:54:44.190345 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:54:44.190297 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:54:47.672506 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:54:47.672124 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:54:48.012881 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:54:48.012793 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:54:48.215249 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:54:48.215205 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:54:50.798258 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:54:50.798220 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:54:55.675103 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:54:55.675062 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:54:59.839430 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:54:59.839394 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:55:00.334452 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:00.334408 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:55:00.603148 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:00.603045 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:55:01.702172 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:01.702133 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:55:04.394912 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:04.394873 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-d8gw2"] Nov 03 01:55:04.395437 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:04.395093 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-d8gw2" podUID="c5ed0a52-1e65-46f3-b268-bdb523bfec88" containerName="discovery" containerID="cri-o://1987b276e5b3148902e698486c70de9265e2b2e6bb9c3b9c85a7f8f2337da888" gracePeriod=30 Nov 03 01:55:04.510467 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:04.510431 5437 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-d8gw2 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.84:8080/ready\": dial tcp 10.134.0.84:8080: connect: connection refused" start-of-body= Nov 03 01:55:04.510639 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:04.510498 5437 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-d8gw2" podUID="c5ed0a52-1e65-46f3-b268-bdb523bfec88" containerName="discovery" probeResult="failure" output="Get \"http://10.134.0.84:8080/ready\": dial tcp 10.134.0.84:8080: connect: connection refused" Nov 03 01:55:04.674207 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:04.674178 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-d8gw2" Nov 03 01:55:04.762628 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:04.761666 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5ed0a52-1e65-46f3-b268-bdb523bfec88-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5ed0a52-1e65-46f3-b268-bdb523bfec88-local-certs\") pod \"c5ed0a52-1e65-46f3-b268-bdb523bfec88\" (UID: \"c5ed0a52-1e65-46f3-b268-bdb523bfec88\") " Nov 03 01:55:04.762628 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:04.761766 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c5ed0a52-1e65-46f3-b268-bdb523bfec88-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c5ed0a52-1e65-46f3-b268-bdb523bfec88-istio-kubeconfig\") pod \"c5ed0a52-1e65-46f3-b268-bdb523bfec88\" (UID: \"c5ed0a52-1e65-46f3-b268-bdb523bfec88\") " Nov 03 01:55:04.762628 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:04.761832 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5ed0a52-1e65-46f3-b268-bdb523bfec88-kube-api-access-gpb6v\" (UniqueName: \"kubernetes.io/projected/c5ed0a52-1e65-46f3-b268-bdb523bfec88-kube-api-access-gpb6v\") pod \"c5ed0a52-1e65-46f3-b268-bdb523bfec88\" (UID: \"c5ed0a52-1e65-46f3-b268-bdb523bfec88\") " Nov 03 01:55:04.762628 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:04.761895 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5ed0a52-1e65-46f3-b268-bdb523bfec88-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c5ed0a52-1e65-46f3-b268-bdb523bfec88-istio-csr-ca-configmap\") pod \"c5ed0a52-1e65-46f3-b268-bdb523bfec88\" (UID: \"c5ed0a52-1e65-46f3-b268-bdb523bfec88\") " Nov 03 01:55:04.762628 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:04.762033 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c5ed0a52-1e65-46f3-b268-bdb523bfec88-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c5ed0a52-1e65-46f3-b268-bdb523bfec88-istio-csr-dns-cert\") pod \"c5ed0a52-1e65-46f3-b268-bdb523bfec88\" (UID: \"c5ed0a52-1e65-46f3-b268-bdb523bfec88\") " Nov 03 01:55:04.762628 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:04.762060 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c5ed0a52-1e65-46f3-b268-bdb523bfec88-cacerts\" (UniqueName: \"kubernetes.io/secret/c5ed0a52-1e65-46f3-b268-bdb523bfec88-cacerts\") pod \"c5ed0a52-1e65-46f3-b268-bdb523bfec88\" (UID: \"c5ed0a52-1e65-46f3-b268-bdb523bfec88\") " Nov 03 01:55:04.762628 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:04.762106 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5ed0a52-1e65-46f3-b268-bdb523bfec88-istio-token\" (UniqueName: \"kubernetes.io/projected/c5ed0a52-1e65-46f3-b268-bdb523bfec88-istio-token\") pod \"c5ed0a52-1e65-46f3-b268-bdb523bfec88\" (UID: \"c5ed0a52-1e65-46f3-b268-bdb523bfec88\") " Nov 03 01:55:04.763143 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:04.762930 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5ed0a52-1e65-46f3-b268-bdb523bfec88-istio-csr-ca-configmap" pod "c5ed0a52-1e65-46f3-b268-bdb523bfec88" (UID: "c5ed0a52-1e65-46f3-b268-bdb523bfec88"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:55:04.764882 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:04.764838 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5ed0a52-1e65-46f3-b268-bdb523bfec88-local-certs" pod "c5ed0a52-1e65-46f3-b268-bdb523bfec88" (UID: "c5ed0a52-1e65-46f3-b268-bdb523bfec88"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:55:04.765756 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:04.765728 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5ed0a52-1e65-46f3-b268-bdb523bfec88-kube-api-access-gpb6v" pod "c5ed0a52-1e65-46f3-b268-bdb523bfec88" (UID: "c5ed0a52-1e65-46f3-b268-bdb523bfec88"). InnerVolumeSpecName "kube-api-access-gpb6v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:55:04.765873 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:04.765819 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5ed0a52-1e65-46f3-b268-bdb523bfec88-istio-csr-dns-cert" pod "c5ed0a52-1e65-46f3-b268-bdb523bfec88" (UID: "c5ed0a52-1e65-46f3-b268-bdb523bfec88"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:55:04.767047 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:04.767009 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5ed0a52-1e65-46f3-b268-bdb523bfec88-cacerts" pod "c5ed0a52-1e65-46f3-b268-bdb523bfec88" (UID: "c5ed0a52-1e65-46f3-b268-bdb523bfec88"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:55:04.767711 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:04.767684 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5ed0a52-1e65-46f3-b268-bdb523bfec88-istio-token" pod "c5ed0a52-1e65-46f3-b268-bdb523bfec88" (UID: "c5ed0a52-1e65-46f3-b268-bdb523bfec88"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:55:04.769072 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:04.769038 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5ed0a52-1e65-46f3-b268-bdb523bfec88-istio-kubeconfig" pod "c5ed0a52-1e65-46f3-b268-bdb523bfec88" (UID: "c5ed0a52-1e65-46f3-b268-bdb523bfec88"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:55:04.778390 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:04.778342 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:55:04.863811 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:04.863765 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpb6v\" (UniqueName: \"kubernetes.io/projected/c5ed0a52-1e65-46f3-b268-bdb523bfec88-kube-api-access-gpb6v\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:55:04.863811 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:04.863812 5437 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c5ed0a52-1e65-46f3-b268-bdb523bfec88-istio-csr-ca-configmap\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:55:04.864083 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:04.863830 5437 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c5ed0a52-1e65-46f3-b268-bdb523bfec88-istio-csr-dns-cert\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:55:04.864083 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:04.863845 5437 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c5ed0a52-1e65-46f3-b268-bdb523bfec88-cacerts\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:55:04.864083 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:04.863859 5437 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5ed0a52-1e65-46f3-b268-bdb523bfec88-istio-token\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:55:04.864083 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:04.863875 5437 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5ed0a52-1e65-46f3-b268-bdb523bfec88-local-certs\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:55:04.864083 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:04.863887 5437 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c5ed0a52-1e65-46f3-b268-bdb523bfec88-istio-kubeconfig\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:55:05.032686 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:05.032594 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:55:05.102294 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:05.102258 5437 generic.go:356] "Generic (PLEG): container finished" podID="c5ed0a52-1e65-46f3-b268-bdb523bfec88" containerID="1987b276e5b3148902e698486c70de9265e2b2e6bb9c3b9c85a7f8f2337da888" exitCode=0 Nov 03 01:55:05.102483 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:05.102311 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-d8gw2" Nov 03 01:55:05.102483 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:05.102325 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-d8gw2" event={"ID":"c5ed0a52-1e65-46f3-b268-bdb523bfec88","Type":"ContainerDied","Data":"1987b276e5b3148902e698486c70de9265e2b2e6bb9c3b9c85a7f8f2337da888"} Nov 03 01:55:05.102483 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:05.102405 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-d8gw2" event={"ID":"c5ed0a52-1e65-46f3-b268-bdb523bfec88","Type":"ContainerDied","Data":"bac2f5feb0a32247ed25ffb61b43231dc9243d959ee4782a7ab4889104244e2c"} Nov 03 01:55:05.102483 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:05.102436 5437 scope.go:122] "RemoveContainer" containerID="1987b276e5b3148902e698486c70de9265e2b2e6bb9c3b9c85a7f8f2337da888" Nov 03 01:55:05.113753 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:05.113729 5437 scope.go:122] "RemoveContainer" containerID="1987b276e5b3148902e698486c70de9265e2b2e6bb9c3b9c85a7f8f2337da888" Nov 03 01:55:05.114093 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:55:05.114073 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1987b276e5b3148902e698486c70de9265e2b2e6bb9c3b9c85a7f8f2337da888\": container with ID starting with 1987b276e5b3148902e698486c70de9265e2b2e6bb9c3b9c85a7f8f2337da888 not found: ID does not exist" containerID="1987b276e5b3148902e698486c70de9265e2b2e6bb9c3b9c85a7f8f2337da888" Nov 03 01:55:05.114142 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:05.114104 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1987b276e5b3148902e698486c70de9265e2b2e6bb9c3b9c85a7f8f2337da888"} err="failed to get container status \"1987b276e5b3148902e698486c70de9265e2b2e6bb9c3b9c85a7f8f2337da888\": rpc error: code = NotFound desc = could not find container \"1987b276e5b3148902e698486c70de9265e2b2e6bb9c3b9c85a7f8f2337da888\": container with ID starting with 1987b276e5b3148902e698486c70de9265e2b2e6bb9c3b9c85a7f8f2337da888 not found: ID does not exist" Nov 03 01:55:05.232171 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:05.232130 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:55:05.274638 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:05.274604 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-d8gw2"] Nov 03 01:55:05.280285 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:05.280255 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-d8gw2"] Nov 03 01:55:06.093812 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:06.093771 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5ed0a52-1e65-46f3-b268-bdb523bfec88" path="/var/lib/kubelet/pods/c5ed0a52-1e65-46f3-b268-bdb523bfec88/volumes" Nov 03 01:55:06.160166 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:06.160128 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:55:06.753407 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:06.753337 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:55:07.263215 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:07.263160 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:55:07.927379 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:07.927316 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:55:08.141160 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:08.141112 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:55:10.962573 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:10.962538 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:55:12.923120 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:12.923068 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:55:13.627996 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:13.627958 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:55:13.873758 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:13.873717 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:55:14.036270 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:55:14.036183 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 01:55:14.721494 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:14.721458 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:55:15.253109 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:15.253067 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:55:15.770883 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:15.770776 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:55:16.657801 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:16.657767 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:55:17.951771 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:17.951724 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:55:21.981574 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:21.981533 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0e0d61-6d6d98697f-x9w8p"] Nov 03 01:55:22.094340 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:22.094303 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-71746a-775978fc79-vbnnk"] Nov 03 01:55:22.094563 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:22.094527 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-71746a-775978fc79-vbnnk" podUID="9a38ff04-2bef-4dc5-8e95-ae1356bf16f4" containerName="maas-api" containerID="cri-o://fce255a63795568ea97fb41c10b880bb5e0ccbd016dd91d55716851c77aff336" gracePeriod=30 Nov 03 01:55:22.199455 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:22.199414 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:55:22.392740 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:22.392711 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-71746a-775978fc79-vbnnk" Nov 03 01:55:22.453303 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:22.453262 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:55:22.479410 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:22.479338 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a38ff04-2bef-4dc5-8e95-ae1356bf16f4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a38ff04-2bef-4dc5-8e95-ae1356bf16f4-metrics-certs\") pod \"9a38ff04-2bef-4dc5-8e95-ae1356bf16f4\" (UID: \"9a38ff04-2bef-4dc5-8e95-ae1356bf16f4\") " Nov 03 01:55:22.479567 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:22.479467 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a38ff04-2bef-4dc5-8e95-ae1356bf16f4-kube-api-access-nwn5r\" (UniqueName: \"kubernetes.io/projected/9a38ff04-2bef-4dc5-8e95-ae1356bf16f4-kube-api-access-nwn5r\") pod \"9a38ff04-2bef-4dc5-8e95-ae1356bf16f4\" (UID: \"9a38ff04-2bef-4dc5-8e95-ae1356bf16f4\") " Nov 03 01:55:22.479567 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:22.479492 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a38ff04-2bef-4dc5-8e95-ae1356bf16f4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a38ff04-2bef-4dc5-8e95-ae1356bf16f4-maas-api-tls\") pod \"9a38ff04-2bef-4dc5-8e95-ae1356bf16f4\" (UID: \"9a38ff04-2bef-4dc5-8e95-ae1356bf16f4\") " Nov 03 01:55:22.479567 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:22.479524 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a38ff04-2bef-4dc5-8e95-ae1356bf16f4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a38ff04-2bef-4dc5-8e95-ae1356bf16f4-openshift-service-ca\") pod \"9a38ff04-2bef-4dc5-8e95-ae1356bf16f4\" (UID: \"9a38ff04-2bef-4dc5-8e95-ae1356bf16f4\") " Nov 03 01:55:22.479959 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:22.479930 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a38ff04-2bef-4dc5-8e95-ae1356bf16f4-openshift-service-ca" pod "9a38ff04-2bef-4dc5-8e95-ae1356bf16f4" (UID: "9a38ff04-2bef-4dc5-8e95-ae1356bf16f4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:55:22.480133 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:22.480105 5437 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a38ff04-2bef-4dc5-8e95-ae1356bf16f4-openshift-service-ca\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:55:22.481723 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:22.481690 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a38ff04-2bef-4dc5-8e95-ae1356bf16f4-kube-api-access-nwn5r" pod "9a38ff04-2bef-4dc5-8e95-ae1356bf16f4" (UID: "9a38ff04-2bef-4dc5-8e95-ae1356bf16f4"). InnerVolumeSpecName "kube-api-access-nwn5r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:55:22.481723 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:22.481713 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a38ff04-2bef-4dc5-8e95-ae1356bf16f4-maas-api-tls" pod "9a38ff04-2bef-4dc5-8e95-ae1356bf16f4" (UID: "9a38ff04-2bef-4dc5-8e95-ae1356bf16f4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:55:22.481887 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:22.481762 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a38ff04-2bef-4dc5-8e95-ae1356bf16f4-metrics-certs" pod "9a38ff04-2bef-4dc5-8e95-ae1356bf16f4" (UID: "9a38ff04-2bef-4dc5-8e95-ae1356bf16f4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:55:22.581427 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:22.581348 5437 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a38ff04-2bef-4dc5-8e95-ae1356bf16f4-metrics-certs\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:55:22.581427 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:22.581422 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwn5r\" (UniqueName: \"kubernetes.io/projected/9a38ff04-2bef-4dc5-8e95-ae1356bf16f4-kube-api-access-nwn5r\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:55:22.581427 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:22.581434 5437 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a38ff04-2bef-4dc5-8e95-ae1356bf16f4-maas-api-tls\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:55:22.799440 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:22.799398 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:55:23.001868 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:23.001754 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:55:23.225327 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:23.222948 5437 generic.go:356] "Generic (PLEG): container finished" podID="9a38ff04-2bef-4dc5-8e95-ae1356bf16f4" containerID="fce255a63795568ea97fb41c10b880bb5e0ccbd016dd91d55716851c77aff336" exitCode=0 Nov 03 01:55:23.225327 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:23.223015 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-71746a-775978fc79-vbnnk" Nov 03 01:55:23.225327 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:23.223158 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-71746a-775978fc79-vbnnk" event={"ID":"9a38ff04-2bef-4dc5-8e95-ae1356bf16f4","Type":"ContainerDied","Data":"fce255a63795568ea97fb41c10b880bb5e0ccbd016dd91d55716851c77aff336"} Nov 03 01:55:23.225327 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:23.223197 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-71746a-775978fc79-vbnnk" event={"ID":"9a38ff04-2bef-4dc5-8e95-ae1356bf16f4","Type":"ContainerDied","Data":"4fe272b4a3d4100bd2d7bbf620eea1b2dcb1f259b6d96f1276097d8c0115775f"} Nov 03 01:55:23.225327 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:23.223224 5437 scope.go:122] "RemoveContainer" containerID="fce255a63795568ea97fb41c10b880bb5e0ccbd016dd91d55716851c77aff336" Nov 03 01:55:23.240860 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:23.240821 5437 scope.go:122] "RemoveContainer" containerID="fce255a63795568ea97fb41c10b880bb5e0ccbd016dd91d55716851c77aff336" Nov 03 01:55:23.241337 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:55:23.241307 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fce255a63795568ea97fb41c10b880bb5e0ccbd016dd91d55716851c77aff336\": container with ID starting with fce255a63795568ea97fb41c10b880bb5e0ccbd016dd91d55716851c77aff336 not found: ID does not exist" containerID="fce255a63795568ea97fb41c10b880bb5e0ccbd016dd91d55716851c77aff336" Nov 03 01:55:23.241494 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:23.241347 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fce255a63795568ea97fb41c10b880bb5e0ccbd016dd91d55716851c77aff336"} err="failed to get container status \"fce255a63795568ea97fb41c10b880bb5e0ccbd016dd91d55716851c77aff336\": rpc error: code = NotFound desc = could not find container \"fce255a63795568ea97fb41c10b880bb5e0ccbd016dd91d55716851c77aff336\": container with ID starting with fce255a63795568ea97fb41c10b880bb5e0ccbd016dd91d55716851c77aff336 not found: ID does not exist" Nov 03 01:55:23.420063 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:23.396808 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-71746a-775978fc79-vbnnk"] Nov 03 01:55:23.420063 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:23.401033 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-71746a-775978fc79-vbnnk"] Nov 03 01:55:23.674407 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:23.674126 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:55:24.090678 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:24.090645 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a38ff04-2bef-4dc5-8e95-ae1356bf16f4" path="/var/lib/kubelet/pods/9a38ff04-2bef-4dc5-8e95-ae1356bf16f4/volumes" Nov 03 01:55:24.170617 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:24.170581 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:55:26.982497 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:26.982424 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0e0d61-6d6d98697f-x9w8p" podUID="43733d32-180d-4918-a97e-0e36ac3405da" containerName="payload-pre-processing" containerID="cri-o://b2eb1ee052c6e17fc026c9c31745407ec2b1e92038edf8c30c6cd9f45014bbe9" gracePeriod=25 Nov 03 01:55:27.243882 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:27.243811 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0e0d61-6d6d98697f-x9w8p" Nov 03 01:55:27.249849 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:27.249812 5437 generic.go:356] "Generic (PLEG): container finished" podID="43733d32-180d-4918-a97e-0e36ac3405da" containerID="b2eb1ee052c6e17fc026c9c31745407ec2b1e92038edf8c30c6cd9f45014bbe9" exitCode=0 Nov 03 01:55:27.250026 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:27.249856 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0e0d61-6d6d98697f-x9w8p" Nov 03 01:55:27.250026 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:27.249894 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0e0d61-6d6d98697f-x9w8p" event={"ID":"43733d32-180d-4918-a97e-0e36ac3405da","Type":"ContainerDied","Data":"b2eb1ee052c6e17fc026c9c31745407ec2b1e92038edf8c30c6cd9f45014bbe9"} Nov 03 01:55:27.250026 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:27.249963 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0e0d61-6d6d98697f-x9w8p" event={"ID":"43733d32-180d-4918-a97e-0e36ac3405da","Type":"ContainerDied","Data":"36fd2fb2abb564b56aecc7626028842ff9b7f1b5ea890880fb20aa4383e7a5ca"} Nov 03 01:55:27.250026 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:27.249993 5437 scope.go:122] "RemoveContainer" containerID="b2eb1ee052c6e17fc026c9c31745407ec2b1e92038edf8c30c6cd9f45014bbe9" Nov 03 01:55:27.261321 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:27.261294 5437 scope.go:122] "RemoveContainer" containerID="b2eb1ee052c6e17fc026c9c31745407ec2b1e92038edf8c30c6cd9f45014bbe9" Nov 03 01:55:27.261686 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:55:27.261659 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2eb1ee052c6e17fc026c9c31745407ec2b1e92038edf8c30c6cd9f45014bbe9\": container with ID starting with b2eb1ee052c6e17fc026c9c31745407ec2b1e92038edf8c30c6cd9f45014bbe9 not found: ID does not exist" containerID="b2eb1ee052c6e17fc026c9c31745407ec2b1e92038edf8c30c6cd9f45014bbe9" Nov 03 01:55:27.261749 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:27.261696 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2eb1ee052c6e17fc026c9c31745407ec2b1e92038edf8c30c6cd9f45014bbe9"} err="failed to get container status \"b2eb1ee052c6e17fc026c9c31745407ec2b1e92038edf8c30c6cd9f45014bbe9\": rpc error: code = NotFound desc = could not find container \"b2eb1ee052c6e17fc026c9c31745407ec2b1e92038edf8c30c6cd9f45014bbe9\": container with ID starting with b2eb1ee052c6e17fc026c9c31745407ec2b1e92038edf8c30c6cd9f45014bbe9 not found: ID does not exist" Nov 03 01:55:27.365935 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:27.365898 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43733d32-180d-4918-a97e-0e36ac3405da-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43733d32-180d-4918-a97e-0e36ac3405da-plugins-config-volume\") pod \"43733d32-180d-4918-a97e-0e36ac3405da\" (UID: \"43733d32-180d-4918-a97e-0e36ac3405da\") " Nov 03 01:55:27.366147 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:27.366010 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43733d32-180d-4918-a97e-0e36ac3405da-kube-api-access-ck6w2\" (UniqueName: \"kubernetes.io/projected/43733d32-180d-4918-a97e-0e36ac3405da-kube-api-access-ck6w2\") pod \"43733d32-180d-4918-a97e-0e36ac3405da\" (UID: \"43733d32-180d-4918-a97e-0e36ac3405da\") " Nov 03 01:55:27.366391 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:27.366340 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43733d32-180d-4918-a97e-0e36ac3405da-plugins-config-volume" pod "43733d32-180d-4918-a97e-0e36ac3405da" (UID: "43733d32-180d-4918-a97e-0e36ac3405da"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:55:27.366611 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:27.366591 5437 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43733d32-180d-4918-a97e-0e36ac3405da-plugins-config-volume\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:55:27.368470 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:27.368439 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43733d32-180d-4918-a97e-0e36ac3405da-kube-api-access-ck6w2" pod "43733d32-180d-4918-a97e-0e36ac3405da" (UID: "43733d32-180d-4918-a97e-0e36ac3405da"). InnerVolumeSpecName "kube-api-access-ck6w2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:55:27.468255 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:27.468216 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ck6w2\" (UniqueName: \"kubernetes.io/projected/43733d32-180d-4918-a97e-0e36ac3405da-kube-api-access-ck6w2\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:55:27.675027 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:27.674990 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0e0d61-6d6d98697f-x9w8p"] Nov 03 01:55:27.678610 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:27.678579 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0e0d61-6d6d98697f-x9w8p"] Nov 03 01:55:28.089066 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:28.089025 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43733d32-180d-4918-a97e-0e36ac3405da" path="/var/lib/kubelet/pods/43733d32-180d-4918-a97e-0e36ac3405da/volumes" Nov 03 01:55:28.830305 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:28.830266 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:55:29.315425 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:29.315346 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:55:29.568622 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:29.568536 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:55:29.768510 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:29.768474 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:55:29.967810 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:29.967708 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:55:30.326467 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:30.326433 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:55:30.547588 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:30.547545 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:55:30.753762 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:30.753668 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:55:32.072627 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:32.072582 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:55:32.589592 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:32.589561 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:55:34.210325 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:34.210294 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:55:35.028945 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:35.028893 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:55:35.234127 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:35.234085 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:55:35.588124 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:35.588079 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:55:35.861995 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:35.861901 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:55:36.127071 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:36.126979 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:55:37.059069 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:37.059034 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:55:37.557539 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:37.557500 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:55:38.381322 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:38.381291 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:55:39.825582 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:39.825540 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:55:40.034440 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:40.034397 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:55:40.364968 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:40.364928 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:55:40.617810 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:40.617720 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:55:41.538746 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:41.538705 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:55:42.962214 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:42.962177 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:55:43.179208 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:43.179175 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:55:44.034582 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:55:44.034547 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 01:55:44.830019 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:44.829976 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:55:47.472179 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:47.472146 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b37ba5-d668d8449-xqgrq"] Nov 03 01:55:47.472640 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:47.472359 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b37ba5-d668d8449-xqgrq" podUID="8282fcb6-7612-4ade-9386-3b1e7ae11b13" containerName="maas-api" containerID="cri-o://1602df86bb1babdd2570e8b02e9b5f30354020a19bff31b4e0cb67221fd25f85" gracePeriod=30 Nov 03 01:55:47.641453 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:47.641412 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:55:47.744882 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:47.744854 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b37ba5-d668d8449-xqgrq" Nov 03 01:55:47.823263 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:47.823232 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8282fcb6-7612-4ade-9386-3b1e7ae11b13-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8282fcb6-7612-4ade-9386-3b1e7ae11b13-maas-api-tls\") pod \"8282fcb6-7612-4ade-9386-3b1e7ae11b13\" (UID: \"8282fcb6-7612-4ade-9386-3b1e7ae11b13\") " Nov 03 01:55:47.823448 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:47.823423 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8282fcb6-7612-4ade-9386-3b1e7ae11b13-kube-api-access-pnzft\" (UniqueName: \"kubernetes.io/projected/8282fcb6-7612-4ade-9386-3b1e7ae11b13-kube-api-access-pnzft\") pod \"8282fcb6-7612-4ade-9386-3b1e7ae11b13\" (UID: \"8282fcb6-7612-4ade-9386-3b1e7ae11b13\") " Nov 03 01:55:47.823511 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:47.823455 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8282fcb6-7612-4ade-9386-3b1e7ae11b13-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8282fcb6-7612-4ade-9386-3b1e7ae11b13-openshift-service-ca\") pod \"8282fcb6-7612-4ade-9386-3b1e7ae11b13\" (UID: \"8282fcb6-7612-4ade-9386-3b1e7ae11b13\") " Nov 03 01:55:47.823511 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:47.823492 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8282fcb6-7612-4ade-9386-3b1e7ae11b13-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8282fcb6-7612-4ade-9386-3b1e7ae11b13-metrics-certs\") pod \"8282fcb6-7612-4ade-9386-3b1e7ae11b13\" (UID: \"8282fcb6-7612-4ade-9386-3b1e7ae11b13\") " Nov 03 01:55:47.823870 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:47.823837 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8282fcb6-7612-4ade-9386-3b1e7ae11b13-openshift-service-ca" pod "8282fcb6-7612-4ade-9386-3b1e7ae11b13" (UID: "8282fcb6-7612-4ade-9386-3b1e7ae11b13"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:55:47.825467 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:47.825440 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8282fcb6-7612-4ade-9386-3b1e7ae11b13-maas-api-tls" pod "8282fcb6-7612-4ade-9386-3b1e7ae11b13" (UID: "8282fcb6-7612-4ade-9386-3b1e7ae11b13"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:55:47.825597 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:47.825575 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8282fcb6-7612-4ade-9386-3b1e7ae11b13-metrics-certs" pod "8282fcb6-7612-4ade-9386-3b1e7ae11b13" (UID: "8282fcb6-7612-4ade-9386-3b1e7ae11b13"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:55:47.825659 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:47.825584 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8282fcb6-7612-4ade-9386-3b1e7ae11b13-kube-api-access-pnzft" pod "8282fcb6-7612-4ade-9386-3b1e7ae11b13" (UID: "8282fcb6-7612-4ade-9386-3b1e7ae11b13"). InnerVolumeSpecName "kube-api-access-pnzft". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:55:47.924905 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:47.924872 5437 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8282fcb6-7612-4ade-9386-3b1e7ae11b13-maas-api-tls\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:55:47.924905 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:47.924901 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnzft\" (UniqueName: \"kubernetes.io/projected/8282fcb6-7612-4ade-9386-3b1e7ae11b13-kube-api-access-pnzft\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:55:47.924905 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:47.924911 5437 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8282fcb6-7612-4ade-9386-3b1e7ae11b13-openshift-service-ca\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:55:47.925140 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:47.924921 5437 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8282fcb6-7612-4ade-9386-3b1e7ae11b13-metrics-certs\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:55:48.122736 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:48.122701 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:55:48.323971 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:48.323933 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:55:48.400321 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:48.400229 5437 generic.go:356] "Generic (PLEG): container finished" podID="8282fcb6-7612-4ade-9386-3b1e7ae11b13" containerID="1602df86bb1babdd2570e8b02e9b5f30354020a19bff31b4e0cb67221fd25f85" exitCode=0 Nov 03 01:55:48.400321 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:48.400281 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b37ba5-d668d8449-xqgrq" Nov 03 01:55:48.400321 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:48.400294 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b37ba5-d668d8449-xqgrq" event={"ID":"8282fcb6-7612-4ade-9386-3b1e7ae11b13","Type":"ContainerDied","Data":"1602df86bb1babdd2570e8b02e9b5f30354020a19bff31b4e0cb67221fd25f85"} Nov 03 01:55:48.400620 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:48.400394 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b37ba5-d668d8449-xqgrq" event={"ID":"8282fcb6-7612-4ade-9386-3b1e7ae11b13","Type":"ContainerDied","Data":"9d76bc86132e0ee825496dcc320b8ca5da20acfcaf5c7af5bff0cd2d7611851b"} Nov 03 01:55:48.400620 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:48.400423 5437 scope.go:122] "RemoveContainer" containerID="1602df86bb1babdd2570e8b02e9b5f30354020a19bff31b4e0cb67221fd25f85" Nov 03 01:55:48.411231 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:48.411199 5437 scope.go:122] "RemoveContainer" containerID="1602df86bb1babdd2570e8b02e9b5f30354020a19bff31b4e0cb67221fd25f85" Nov 03 01:55:48.411565 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:55:48.411544 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1602df86bb1babdd2570e8b02e9b5f30354020a19bff31b4e0cb67221fd25f85\": container with ID starting with 1602df86bb1babdd2570e8b02e9b5f30354020a19bff31b4e0cb67221fd25f85 not found: ID does not exist" containerID="1602df86bb1babdd2570e8b02e9b5f30354020a19bff31b4e0cb67221fd25f85" Nov 03 01:55:48.411642 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:48.411573 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1602df86bb1babdd2570e8b02e9b5f30354020a19bff31b4e0cb67221fd25f85"} err="failed to get container status \"1602df86bb1babdd2570e8b02e9b5f30354020a19bff31b4e0cb67221fd25f85\": rpc error: code = NotFound desc = could not find container \"1602df86bb1babdd2570e8b02e9b5f30354020a19bff31b4e0cb67221fd25f85\": container with ID starting with 1602df86bb1babdd2570e8b02e9b5f30354020a19bff31b4e0cb67221fd25f85 not found: ID does not exist" Nov 03 01:55:48.528259 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:48.528221 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:55:48.572344 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:48.572315 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b37ba5-d668d8449-xqgrq"] Nov 03 01:55:48.576964 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:48.576938 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b37ba5-d668d8449-xqgrq"] Nov 03 01:55:49.533139 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:49.533096 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:55:50.088324 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:50.088280 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8282fcb6-7612-4ade-9386-3b1e7ae11b13" path="/var/lib/kubelet/pods/8282fcb6-7612-4ade-9386-3b1e7ae11b13/volumes" Nov 03 01:55:50.437565 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:50.437444 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:55:50.641699 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:50.641663 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:55:51.958580 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:51.958541 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:55:52.445830 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:52.445779 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:55:54.697428 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:54.697393 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:55:58.359700 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:58.359658 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:55:58.860530 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:55:58.860494 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:56:01.578668 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:01.578626 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:56:05.063808 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:05.063773 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:56:05.664768 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:05.664724 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:56:05.917312 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:05.917211 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:56:06.380285 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:06.380249 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:56:10.536867 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:10.536822 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:56:11.988495 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:11.988456 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:56:12.244697 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:12.244604 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:56:12.483908 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:12.483870 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:56:13.512783 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:13.512744 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:56:14.024881 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:14.024841 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:56:14.035220 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:56:14.035186 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 01:56:15.017236 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:15.017198 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:56:15.708046 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:15.708013 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:56:15.959993 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:15.959894 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:56:17.367396 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:17.367131 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:56:22.375965 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:22.375929 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:56:27.180226 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:27.180187 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:56:28.011837 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:28.011792 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:56:29.309399 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:29.309350 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:56:30.011222 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:30.011180 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:56:30.338139 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:30.338102 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:56:32.116702 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:32.116663 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:56:32.321897 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:32.321863 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:56:36.473883 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:36.473846 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:56:36.673935 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:36.673902 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:56:37.264811 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:37.264772 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:56:40.402282 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:40.402245 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:56:40.610293 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:40.610254 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:56:41.013756 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:41.013716 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:56:41.226064 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:41.226027 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:56:41.583719 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:41.583681 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:56:41.792776 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:41.792727 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:56:43.745030 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:43.744980 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:56:43.994092 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:43.994041 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:56:44.034724 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:56:44.034640 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 01:56:44.096865 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:44.096829 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-e2db44-65b559dccf-d5wgm"] Nov 03 01:56:44.097352 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:44.097340 5437 cpu_manager.go:414] "containerMap: removing container" podUID="c5ed0a52-1e65-46f3-b268-bdb523bfec88" containerName="discovery" Nov 03 01:56:44.097417 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:44.097353 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5ed0a52-1e65-46f3-b268-bdb523bfec88" containerName="discovery" Nov 03 01:56:44.097417 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:44.097380 5437 cpu_manager.go:414] "containerMap: removing container" podUID="8282fcb6-7612-4ade-9386-3b1e7ae11b13" containerName="maas-api" Nov 03 01:56:44.097417 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:44.097386 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8282fcb6-7612-4ade-9386-3b1e7ae11b13" containerName="maas-api" Nov 03 01:56:44.097417 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:44.097394 5437 cpu_manager.go:414] "containerMap: removing container" podUID="04cd0e8c-c175-4c88-a5d3-856e5c04fe98" containerName="payload-processing" Nov 03 01:56:44.097417 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:44.097399 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04cd0e8c-c175-4c88-a5d3-856e5c04fe98" containerName="payload-processing" Nov 03 01:56:44.097417 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:44.097416 5437 cpu_manager.go:414] "containerMap: removing container" podUID="1aa3545f-2866-4899-8e50-1b55db8a4b1b" containerName="istio-proxy" Nov 03 01:56:44.097603 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:44.097421 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1aa3545f-2866-4899-8e50-1b55db8a4b1b" containerName="istio-proxy" Nov 03 01:56:44.097603 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:44.097431 5437 cpu_manager.go:414] "containerMap: removing container" podUID="d37572bf-d59f-4e9a-8745-f4d91be336f5" containerName="payload-pre-processing" Nov 03 01:56:44.097603 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:44.097436 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d37572bf-d59f-4e9a-8745-f4d91be336f5" containerName="payload-pre-processing" Nov 03 01:56:44.097603 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:44.097448 5437 cpu_manager.go:414] "containerMap: removing container" podUID="43733d32-180d-4918-a97e-0e36ac3405da" containerName="payload-pre-processing" Nov 03 01:56:44.097603 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:44.097452 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43733d32-180d-4918-a97e-0e36ac3405da" containerName="payload-pre-processing" Nov 03 01:56:44.097603 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:44.097466 5437 cpu_manager.go:414] "containerMap: removing container" podUID="9a38ff04-2bef-4dc5-8e95-ae1356bf16f4" containerName="maas-api" Nov 03 01:56:44.097603 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:44.097471 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a38ff04-2bef-4dc5-8e95-ae1356bf16f4" containerName="maas-api" Nov 03 01:56:44.097603 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:44.097549 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="43733d32-180d-4918-a97e-0e36ac3405da" containerName="payload-pre-processing" Nov 03 01:56:44.097603 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:44.097562 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a38ff04-2bef-4dc5-8e95-ae1356bf16f4" containerName="maas-api" Nov 03 01:56:44.097603 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:44.097571 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5ed0a52-1e65-46f3-b268-bdb523bfec88" containerName="discovery" Nov 03 01:56:44.097603 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:44.097577 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="8282fcb6-7612-4ade-9386-3b1e7ae11b13" containerName="maas-api" Nov 03 01:56:44.097603 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:44.097584 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="04cd0e8c-c175-4c88-a5d3-856e5c04fe98" containerName="payload-processing" Nov 03 01:56:44.097603 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:44.097591 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="1aa3545f-2866-4899-8e50-1b55db8a4b1b" containerName="istio-proxy" Nov 03 01:56:44.097603 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:44.097597 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="d37572bf-d59f-4e9a-8745-f4d91be336f5" containerName="payload-pre-processing" Nov 03 01:56:44.101962 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:44.101943 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e2db44-65b559dccf-d5wgm" Nov 03 01:56:44.158811 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:44.158772 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-e2db44-65b559dccf-d5wgm"] Nov 03 01:56:44.179738 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:44.179694 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1375bf2-794e-4412-be74-5bd983bd4ee6-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-e2db44-65b559dccf-d5wgm\" (UID: \"c1375bf2-794e-4412-be74-5bd983bd4ee6\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e2db44-65b559dccf-d5wgm" Nov 03 01:56:44.180511 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:44.180430 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9g7s\" (UniqueName: \"kubernetes.io/projected/c1375bf2-794e-4412-be74-5bd983bd4ee6-kube-api-access-b9g7s\") pod \"payload-pre-processing-e2e-praxis-b-e2db44-65b559dccf-d5wgm\" (UID: \"c1375bf2-794e-4412-be74-5bd983bd4ee6\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e2db44-65b559dccf-d5wgm" Nov 03 01:56:44.198395 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:44.196734 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:56:44.204026 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:44.203994 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-e2db44-dockercfg-4vlsb\"" Nov 03 01:56:44.204026 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:44.203994 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-e2db44\"" Nov 03 01:56:44.281619 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:44.281579 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1375bf2-794e-4412-be74-5bd983bd4ee6-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-e2db44-65b559dccf-d5wgm\" (UID: \"c1375bf2-794e-4412-be74-5bd983bd4ee6\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e2db44-65b559dccf-d5wgm" Nov 03 01:56:44.281838 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:44.281795 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9g7s\" (UniqueName: \"kubernetes.io/projected/c1375bf2-794e-4412-be74-5bd983bd4ee6-kube-api-access-b9g7s\") pod \"payload-pre-processing-e2e-praxis-b-e2db44-65b559dccf-d5wgm\" (UID: \"c1375bf2-794e-4412-be74-5bd983bd4ee6\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e2db44-65b559dccf-d5wgm" Nov 03 01:56:44.282452 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:44.282428 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1375bf2-794e-4412-be74-5bd983bd4ee6-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-e2db44-65b559dccf-d5wgm\" (UID: \"c1375bf2-794e-4412-be74-5bd983bd4ee6\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e2db44-65b559dccf-d5wgm" Nov 03 01:56:44.344507 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:44.344470 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9g7s\" (UniqueName: \"kubernetes.io/projected/c1375bf2-794e-4412-be74-5bd983bd4ee6-kube-api-access-b9g7s\") pod \"payload-pre-processing-e2e-praxis-b-e2db44-65b559dccf-d5wgm\" (UID: \"c1375bf2-794e-4412-be74-5bd983bd4ee6\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e2db44-65b559dccf-d5wgm" Nov 03 01:56:44.415695 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:44.415662 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e2db44-65b559dccf-d5wgm" Nov 03 01:56:44.789321 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:56:44.789290 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1375bf2_794e_4412_be74_5bd983bd4ee6.slice/crio-26ba4c315f0d6e1222996c5026d73d2ccfe125a9c33c2cd22e928eb41f470dc5 WatchSource:0}: Error finding container 26ba4c315f0d6e1222996c5026d73d2ccfe125a9c33c2cd22e928eb41f470dc5: Status 404 returned error can't find the container with id 26ba4c315f0d6e1222996c5026d73d2ccfe125a9c33c2cd22e928eb41f470dc5 Nov 03 01:56:44.792755 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:44.792733 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-e2db44-65b559dccf-d5wgm"] Nov 03 01:56:45.739990 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:45.739959 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e2db44-65b559dccf-d5wgm" event={"ID":"c1375bf2-794e-4412-be74-5bd983bd4ee6","Type":"ContainerStarted","Data":"cd24e1ac0cc70534bed6e836570a2416a0b199c370590f36ccec229600688f8e"} Nov 03 01:56:45.740165 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:45.740007 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e2db44-65b559dccf-d5wgm" event={"ID":"c1375bf2-794e-4412-be74-5bd983bd4ee6","Type":"ContainerStarted","Data":"26ba4c315f0d6e1222996c5026d73d2ccfe125a9c33c2cd22e928eb41f470dc5"} Nov 03 01:56:45.740165 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:45.740105 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e2db44-65b559dccf-d5wgm" Nov 03 01:56:45.859209 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:45.859153 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e2db44-65b559dccf-d5wgm" podStartSLOduration=1.859139414 podStartE2EDuration="1.859139414s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:56:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:56:45.858492475 +0000 UTC m=+2287.887974770" watchObservedRunningTime="2026-11-03 01:56:45.859139414 +0000 UTC m=+2287.888621709" Nov 03 01:56:48.367426 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:48.367391 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:56:48.589321 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:48.589290 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:56:48.701050 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:48.700973 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-70235a-74f5459759-7hw6j"] Nov 03 01:56:48.711499 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:48.711462 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-70235a-74f5459759-7hw6j" Nov 03 01:56:48.759585 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:48.759551 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-70235a-74f5459759-7hw6j"] Nov 03 01:56:48.801698 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:48.800556 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-70235a\"" Nov 03 01:56:48.808148 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:48.808116 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-70235a-dockercfg-gsvfl\"" Nov 03 01:56:48.813718 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:48.812425 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:56:48.844422 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:48.844384 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sk86j\" (UniqueName: \"kubernetes.io/projected/2e18e1b3-1ae6-4f7d-86ab-3821e7d5b72d-kube-api-access-sk86j\") pod \"payload-pre-processing-e2e-praxis-a-70235a-74f5459759-7hw6j\" (UID: \"2e18e1b3-1ae6-4f7d-86ab-3821e7d5b72d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-70235a-74f5459759-7hw6j" Nov 03 01:56:48.844596 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:48.844471 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e18e1b3-1ae6-4f7d-86ab-3821e7d5b72d-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-70235a-74f5459759-7hw6j\" (UID: \"2e18e1b3-1ae6-4f7d-86ab-3821e7d5b72d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-70235a-74f5459759-7hw6j" Nov 03 01:56:48.865035 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:48.865000 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-70235a-6f9d5df8dc-xgzc9"] Nov 03 01:56:48.869271 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:48.869247 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-70235a-6f9d5df8dc-xgzc9" Nov 03 01:56:48.922205 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:48.922173 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-70235a-6f9d5df8dc-xgzc9"] Nov 03 01:56:48.945925 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:48.945889 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sk86j\" (UniqueName: \"kubernetes.io/projected/2e18e1b3-1ae6-4f7d-86ab-3821e7d5b72d-kube-api-access-sk86j\") pod \"payload-pre-processing-e2e-praxis-a-70235a-74f5459759-7hw6j\" (UID: \"2e18e1b3-1ae6-4f7d-86ab-3821e7d5b72d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-70235a-74f5459759-7hw6j" Nov 03 01:56:48.946065 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:48.945929 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e18e1b3-1ae6-4f7d-86ab-3821e7d5b72d-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-70235a-74f5459759-7hw6j\" (UID: \"2e18e1b3-1ae6-4f7d-86ab-3821e7d5b72d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-70235a-74f5459759-7hw6j" Nov 03 01:56:48.946962 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:48.946938 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e18e1b3-1ae6-4f7d-86ab-3821e7d5b72d-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-70235a-74f5459759-7hw6j\" (UID: \"2e18e1b3-1ae6-4f7d-86ab-3821e7d5b72d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-70235a-74f5459759-7hw6j" Nov 03 01:56:49.009359 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:49.009286 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sk86j\" (UniqueName: \"kubernetes.io/projected/2e18e1b3-1ae6-4f7d-86ab-3821e7d5b72d-kube-api-access-sk86j\") pod \"payload-pre-processing-e2e-praxis-a-70235a-74f5459759-7hw6j\" (UID: \"2e18e1b3-1ae6-4f7d-86ab-3821e7d5b72d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-70235a-74f5459759-7hw6j" Nov 03 01:56:49.024724 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:49.024689 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-70235a-74f5459759-7hw6j" Nov 03 01:56:49.047618 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:49.047588 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzj4k\" (UniqueName: \"kubernetes.io/projected/f1e4d46f-f257-45ea-8ffe-10e156db99ed-kube-api-access-qzj4k\") pod \"payload-processing-e2e-praxis-a-70235a-6f9d5df8dc-xgzc9\" (UID: \"f1e4d46f-f257-45ea-8ffe-10e156db99ed\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-70235a-6f9d5df8dc-xgzc9" Nov 03 01:56:49.047783 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:49.047766 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1e4d46f-f257-45ea-8ffe-10e156db99ed-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-70235a-6f9d5df8dc-xgzc9\" (UID: \"f1e4d46f-f257-45ea-8ffe-10e156db99ed\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-70235a-6f9d5df8dc-xgzc9" Nov 03 01:56:49.091341 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:49.091168 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:56:49.148908 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:49.148864 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1e4d46f-f257-45ea-8ffe-10e156db99ed-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-70235a-6f9d5df8dc-xgzc9\" (UID: \"f1e4d46f-f257-45ea-8ffe-10e156db99ed\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-70235a-6f9d5df8dc-xgzc9" Nov 03 01:56:49.149105 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:49.148982 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzj4k\" (UniqueName: \"kubernetes.io/projected/f1e4d46f-f257-45ea-8ffe-10e156db99ed-kube-api-access-qzj4k\") pod \"payload-processing-e2e-praxis-a-70235a-6f9d5df8dc-xgzc9\" (UID: \"f1e4d46f-f257-45ea-8ffe-10e156db99ed\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-70235a-6f9d5df8dc-xgzc9" Nov 03 01:56:49.149859 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:49.149834 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1e4d46f-f257-45ea-8ffe-10e156db99ed-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-70235a-6f9d5df8dc-xgzc9\" (UID: \"f1e4d46f-f257-45ea-8ffe-10e156db99ed\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-70235a-6f9d5df8dc-xgzc9" Nov 03 01:56:49.211676 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:49.211644 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzj4k\" (UniqueName: \"kubernetes.io/projected/f1e4d46f-f257-45ea-8ffe-10e156db99ed-kube-api-access-qzj4k\") pod \"payload-processing-e2e-praxis-a-70235a-6f9d5df8dc-xgzc9\" (UID: \"f1e4d46f-f257-45ea-8ffe-10e156db99ed\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-70235a-6f9d5df8dc-xgzc9" Nov 03 01:56:49.403644 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:56:49.402628 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e18e1b3_1ae6_4f7d_86ab_3821e7d5b72d.slice/crio-31709f0e4c57753bd3e0553b6eba971629cfb527f7596b066d57541ed7dc7930 WatchSource:0}: Error finding container 31709f0e4c57753bd3e0553b6eba971629cfb527f7596b066d57541ed7dc7930: Status 404 returned error can't find the container with id 31709f0e4c57753bd3e0553b6eba971629cfb527f7596b066d57541ed7dc7930 Nov 03 01:56:49.406271 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:49.406240 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-70235a-74f5459759-7hw6j"] Nov 03 01:56:49.483269 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:49.483240 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-70235a-6f9d5df8dc-xgzc9" Nov 03 01:56:49.765128 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:49.765036 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-70235a-74f5459759-7hw6j" event={"ID":"2e18e1b3-1ae6-4f7d-86ab-3821e7d5b72d","Type":"ContainerStarted","Data":"1f19395c3d50d99790a7db5acc59df89015c0dd73b6519052d4f0bfa6681547c"} Nov 03 01:56:49.765128 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:49.765098 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-70235a-74f5459759-7hw6j" event={"ID":"2e18e1b3-1ae6-4f7d-86ab-3821e7d5b72d","Type":"ContainerStarted","Data":"31709f0e4c57753bd3e0553b6eba971629cfb527f7596b066d57541ed7dc7930"} Nov 03 01:56:49.765322 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:49.765241 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-70235a-74f5459759-7hw6j" Nov 03 01:56:49.854055 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:56:49.854023 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1e4d46f_f257_45ea_8ffe_10e156db99ed.slice/crio-13a58cf783e37194b692bccc6a9e46622eafaac4ff91c5cddb014f48bf8c8af8 WatchSource:0}: Error finding container 13a58cf783e37194b692bccc6a9e46622eafaac4ff91c5cddb014f48bf8c8af8: Status 404 returned error can't find the container with id 13a58cf783e37194b692bccc6a9e46622eafaac4ff91c5cddb014f48bf8c8af8 Nov 03 01:56:49.857048 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:49.856991 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-70235a-6f9d5df8dc-xgzc9"] Nov 03 01:56:49.884855 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:49.884805 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-70235a-74f5459759-7hw6j" podStartSLOduration=1.884792414 podStartE2EDuration="1.884792414s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:56:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:56:49.88349139 +0000 UTC m=+2291.912973682" watchObservedRunningTime="2026-11-03 01:56:49.884792414 +0000 UTC m=+2291.914274710" Nov 03 01:56:50.745567 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:50.745529 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:56:50.772725 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:50.772689 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-70235a-6f9d5df8dc-xgzc9" event={"ID":"f1e4d46f-f257-45ea-8ffe-10e156db99ed","Type":"ContainerStarted","Data":"ac8aaab0922763b48fe3601e143bf0be743ef0cf15d88835d7a117c52b717a63"} Nov 03 01:56:50.772912 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:50.772751 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-70235a-6f9d5df8dc-xgzc9" event={"ID":"f1e4d46f-f257-45ea-8ffe-10e156db99ed","Type":"ContainerStarted","Data":"13a58cf783e37194b692bccc6a9e46622eafaac4ff91c5cddb014f48bf8c8af8"} Nov 03 01:56:50.773081 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:50.773051 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-70235a-6f9d5df8dc-xgzc9" Nov 03 01:56:50.892748 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:50.892693 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-70235a-6f9d5df8dc-xgzc9" podStartSLOduration=2.892679923 podStartE2EDuration="2.892679923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:56:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:56:50.891566405 +0000 UTC m=+2292.921048700" watchObservedRunningTime="2026-11-03 01:56:50.892679923 +0000 UTC m=+2292.922162217" Nov 03 01:56:51.001933 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:51.001840 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:56:51.257200 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:51.257117 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f95773-79bdcd49ff-mvz76"] Nov 03 01:56:51.266036 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:51.263926 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f95773-79bdcd49ff-mvz76" Nov 03 01:56:51.314242 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:51.314209 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f95773-79bdcd49ff-mvz76"] Nov 03 01:56:51.363555 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:51.363517 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-f95773\"" Nov 03 01:56:51.363555 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:51.363517 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-f95773-dockercfg-d6885\"" Nov 03 01:56:51.375452 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:51.375423 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgw4h\" (UniqueName: \"kubernetes.io/projected/c31e0981-3a7c-4af3-9108-db5956c52d19-kube-api-access-lgw4h\") pod \"payload-processing-e2e-ipp-a-f95773-79bdcd49ff-mvz76\" (UID: \"c31e0981-3a7c-4af3-9108-db5956c52d19\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f95773-79bdcd49ff-mvz76" Nov 03 01:56:51.375616 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:51.375597 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c31e0981-3a7c-4af3-9108-db5956c52d19-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f95773-79bdcd49ff-mvz76\" (UID: \"c31e0981-3a7c-4af3-9108-db5956c52d19\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f95773-79bdcd49ff-mvz76" Nov 03 01:56:51.476529 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:51.476493 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgw4h\" (UniqueName: \"kubernetes.io/projected/c31e0981-3a7c-4af3-9108-db5956c52d19-kube-api-access-lgw4h\") pod \"payload-processing-e2e-ipp-a-f95773-79bdcd49ff-mvz76\" (UID: \"c31e0981-3a7c-4af3-9108-db5956c52d19\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f95773-79bdcd49ff-mvz76" Nov 03 01:56:51.476706 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:51.476616 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c31e0981-3a7c-4af3-9108-db5956c52d19-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f95773-79bdcd49ff-mvz76\" (UID: \"c31e0981-3a7c-4af3-9108-db5956c52d19\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f95773-79bdcd49ff-mvz76" Nov 03 01:56:51.477306 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:51.477283 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c31e0981-3a7c-4af3-9108-db5956c52d19-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f95773-79bdcd49ff-mvz76\" (UID: \"c31e0981-3a7c-4af3-9108-db5956c52d19\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f95773-79bdcd49ff-mvz76" Nov 03 01:56:51.539217 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:51.539142 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgw4h\" (UniqueName: \"kubernetes.io/projected/c31e0981-3a7c-4af3-9108-db5956c52d19-kube-api-access-lgw4h\") pod \"payload-processing-e2e-ipp-a-f95773-79bdcd49ff-mvz76\" (UID: \"c31e0981-3a7c-4af3-9108-db5956c52d19\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f95773-79bdcd49ff-mvz76" Nov 03 01:56:51.579176 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:51.579140 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f95773-79bdcd49ff-mvz76" Nov 03 01:56:51.952075 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:56:51.952029 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc31e0981_3a7c_4af3_9108_db5956c52d19.slice/crio-d4dc0dbdfdb5c8158d76eb26c79cc32aa0b0d46545c11d2b7334348e5293da37 WatchSource:0}: Error finding container d4dc0dbdfdb5c8158d76eb26c79cc32aa0b0d46545c11d2b7334348e5293da37: Status 404 returned error can't find the container with id d4dc0dbdfdb5c8158d76eb26c79cc32aa0b0d46545c11d2b7334348e5293da37 Nov 03 01:56:51.954638 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:51.954616 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f95773-79bdcd49ff-mvz76"] Nov 03 01:56:52.785839 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:52.785805 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f95773-79bdcd49ff-mvz76" event={"ID":"c31e0981-3a7c-4af3-9108-db5956c52d19","Type":"ContainerStarted","Data":"729a5d1511d7b9871ea37cf11ab0a327066d54deddd04a5ab0977a35f84e5ba7"} Nov 03 01:56:52.786067 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:52.785863 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f95773-79bdcd49ff-mvz76" event={"ID":"c31e0981-3a7c-4af3-9108-db5956c52d19","Type":"ContainerStarted","Data":"d4dc0dbdfdb5c8158d76eb26c79cc32aa0b0d46545c11d2b7334348e5293da37"} Nov 03 01:56:52.786067 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:52.785923 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f95773-79bdcd49ff-mvz76" Nov 03 01:56:52.905478 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:52.905428 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-f95773-79bdcd49ff-mvz76" podStartSLOduration=1.905414676 podStartE2EDuration="1.905414676s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:56:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:56:52.904254203 +0000 UTC m=+2294.933736497" watchObservedRunningTime="2026-11-03 01:56:52.905414676 +0000 UTC m=+2294.934896968" Nov 03 01:56:55.463745 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:55.463708 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:56:56.750005 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:56.749977 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e2db44-65b559dccf-d5wgm" Nov 03 01:56:57.063855 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:57.063819 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-b177c9/ipp-route-b177c9-kserve-7b945c8cfc-gk9vl"] Nov 03 01:56:57.068022 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:57.068001 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-b177c9/ipp-route-b177c9-kserve-7b945c8cfc-gk9vl" Nov 03 01:56:57.122587 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:57.122551 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-b177c9\"/\"kube-root-ca.crt\"" Nov 03 01:56:57.122791 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:57.122669 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-b177c9\"/\"openshift-service-ca.crt\"" Nov 03 01:56:57.124178 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:57.124147 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-b177c9/ipp-route-b177c9-kserve-7b945c8cfc-gk9vl"] Nov 03 01:56:57.138631 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:57.138601 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b29aa32a-3985-4f47-906c-e27b88d21617-tls-certs\") pod \"ipp-route-b177c9-kserve-7b945c8cfc-gk9vl\" (UID: \"b29aa32a-3985-4f47-906c-e27b88d21617\") " pod="ai-tenant-e2e-ipp-a-b177c9/ipp-route-b177c9-kserve-7b945c8cfc-gk9vl" Nov 03 01:56:57.138774 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:57.138655 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2jdz\" (UniqueName: \"kubernetes.io/projected/b29aa32a-3985-4f47-906c-e27b88d21617-kube-api-access-t2jdz\") pod \"ipp-route-b177c9-kserve-7b945c8cfc-gk9vl\" (UID: \"b29aa32a-3985-4f47-906c-e27b88d21617\") " pod="ai-tenant-e2e-ipp-a-b177c9/ipp-route-b177c9-kserve-7b945c8cfc-gk9vl" Nov 03 01:56:57.138823 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:57.138806 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b29aa32a-3985-4f47-906c-e27b88d21617-model-cache\") pod \"ipp-route-b177c9-kserve-7b945c8cfc-gk9vl\" (UID: \"b29aa32a-3985-4f47-906c-e27b88d21617\") " pod="ai-tenant-e2e-ipp-a-b177c9/ipp-route-b177c9-kserve-7b945c8cfc-gk9vl" Nov 03 01:56:57.138980 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:57.138955 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b29aa32a-3985-4f47-906c-e27b88d21617-home\") pod \"ipp-route-b177c9-kserve-7b945c8cfc-gk9vl\" (UID: \"b29aa32a-3985-4f47-906c-e27b88d21617\") " pod="ai-tenant-e2e-ipp-a-b177c9/ipp-route-b177c9-kserve-7b945c8cfc-gk9vl" Nov 03 01:56:57.139128 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:57.139037 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b29aa32a-3985-4f47-906c-e27b88d21617-dshm\") pod \"ipp-route-b177c9-kserve-7b945c8cfc-gk9vl\" (UID: \"b29aa32a-3985-4f47-906c-e27b88d21617\") " pod="ai-tenant-e2e-ipp-a-b177c9/ipp-route-b177c9-kserve-7b945c8cfc-gk9vl" Nov 03 01:56:57.139128 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:57.139082 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b29aa32a-3985-4f47-906c-e27b88d21617-tmp-dir\") pod \"ipp-route-b177c9-kserve-7b945c8cfc-gk9vl\" (UID: \"b29aa32a-3985-4f47-906c-e27b88d21617\") " pod="ai-tenant-e2e-ipp-a-b177c9/ipp-route-b177c9-kserve-7b945c8cfc-gk9vl" Nov 03 01:56:57.172904 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:57.172866 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-b177c9\"/\"default-dockercfg-qm6hq\"" Nov 03 01:56:57.172904 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:57.172902 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-b177c9\"/\"ipp-route-b177c9-kserve-self-signed-certs\"" Nov 03 01:56:57.209191 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:57.209164 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:56:57.240673 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:57.240636 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b29aa32a-3985-4f47-906c-e27b88d21617-tmp-dir\") pod \"ipp-route-b177c9-kserve-7b945c8cfc-gk9vl\" (UID: \"b29aa32a-3985-4f47-906c-e27b88d21617\") " pod="ai-tenant-e2e-ipp-a-b177c9/ipp-route-b177c9-kserve-7b945c8cfc-gk9vl" Nov 03 01:56:57.240869 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:57.240687 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b29aa32a-3985-4f47-906c-e27b88d21617-tls-certs\") pod \"ipp-route-b177c9-kserve-7b945c8cfc-gk9vl\" (UID: \"b29aa32a-3985-4f47-906c-e27b88d21617\") " pod="ai-tenant-e2e-ipp-a-b177c9/ipp-route-b177c9-kserve-7b945c8cfc-gk9vl" Nov 03 01:56:57.240869 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:57.240739 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2jdz\" (UniqueName: \"kubernetes.io/projected/b29aa32a-3985-4f47-906c-e27b88d21617-kube-api-access-t2jdz\") pod \"ipp-route-b177c9-kserve-7b945c8cfc-gk9vl\" (UID: \"b29aa32a-3985-4f47-906c-e27b88d21617\") " pod="ai-tenant-e2e-ipp-a-b177c9/ipp-route-b177c9-kserve-7b945c8cfc-gk9vl" Nov 03 01:56:57.240869 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:57.240810 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b29aa32a-3985-4f47-906c-e27b88d21617-model-cache\") pod \"ipp-route-b177c9-kserve-7b945c8cfc-gk9vl\" (UID: \"b29aa32a-3985-4f47-906c-e27b88d21617\") " pod="ai-tenant-e2e-ipp-a-b177c9/ipp-route-b177c9-kserve-7b945c8cfc-gk9vl" Nov 03 01:56:57.241083 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:57.240883 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b29aa32a-3985-4f47-906c-e27b88d21617-home\") pod \"ipp-route-b177c9-kserve-7b945c8cfc-gk9vl\" (UID: \"b29aa32a-3985-4f47-906c-e27b88d21617\") " pod="ai-tenant-e2e-ipp-a-b177c9/ipp-route-b177c9-kserve-7b945c8cfc-gk9vl" Nov 03 01:56:57.241083 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:57.240926 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b29aa32a-3985-4f47-906c-e27b88d21617-dshm\") pod \"ipp-route-b177c9-kserve-7b945c8cfc-gk9vl\" (UID: \"b29aa32a-3985-4f47-906c-e27b88d21617\") " pod="ai-tenant-e2e-ipp-a-b177c9/ipp-route-b177c9-kserve-7b945c8cfc-gk9vl" Nov 03 01:56:57.241193 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:57.241107 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b29aa32a-3985-4f47-906c-e27b88d21617-tmp-dir\") pod \"ipp-route-b177c9-kserve-7b945c8cfc-gk9vl\" (UID: \"b29aa32a-3985-4f47-906c-e27b88d21617\") " pod="ai-tenant-e2e-ipp-a-b177c9/ipp-route-b177c9-kserve-7b945c8cfc-gk9vl" Nov 03 01:56:57.241193 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:57.241146 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b29aa32a-3985-4f47-906c-e27b88d21617-model-cache\") pod \"ipp-route-b177c9-kserve-7b945c8cfc-gk9vl\" (UID: \"b29aa32a-3985-4f47-906c-e27b88d21617\") " pod="ai-tenant-e2e-ipp-a-b177c9/ipp-route-b177c9-kserve-7b945c8cfc-gk9vl" Nov 03 01:56:57.241326 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:57.241300 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b29aa32a-3985-4f47-906c-e27b88d21617-home\") pod \"ipp-route-b177c9-kserve-7b945c8cfc-gk9vl\" (UID: \"b29aa32a-3985-4f47-906c-e27b88d21617\") " pod="ai-tenant-e2e-ipp-a-b177c9/ipp-route-b177c9-kserve-7b945c8cfc-gk9vl" Nov 03 01:56:57.243130 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:57.243108 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b29aa32a-3985-4f47-906c-e27b88d21617-dshm\") pod \"ipp-route-b177c9-kserve-7b945c8cfc-gk9vl\" (UID: \"b29aa32a-3985-4f47-906c-e27b88d21617\") " pod="ai-tenant-e2e-ipp-a-b177c9/ipp-route-b177c9-kserve-7b945c8cfc-gk9vl" Nov 03 01:56:57.243286 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:57.243269 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b29aa32a-3985-4f47-906c-e27b88d21617-tls-certs\") pod \"ipp-route-b177c9-kserve-7b945c8cfc-gk9vl\" (UID: \"b29aa32a-3985-4f47-906c-e27b88d21617\") " pod="ai-tenant-e2e-ipp-a-b177c9/ipp-route-b177c9-kserve-7b945c8cfc-gk9vl" Nov 03 01:56:57.305583 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:57.305553 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2jdz\" (UniqueName: \"kubernetes.io/projected/b29aa32a-3985-4f47-906c-e27b88d21617-kube-api-access-t2jdz\") pod \"ipp-route-b177c9-kserve-7b945c8cfc-gk9vl\" (UID: \"b29aa32a-3985-4f47-906c-e27b88d21617\") " pod="ai-tenant-e2e-ipp-a-b177c9/ipp-route-b177c9-kserve-7b945c8cfc-gk9vl" Nov 03 01:56:57.380584 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:57.380505 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-b177c9/ipp-route-b177c9-kserve-7b945c8cfc-gk9vl" Nov 03 01:56:57.753634 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:56:57.753603 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb29aa32a_3985_4f47_906c_e27b88d21617.slice/crio-46b4652eb3fc84d3d146115a78f81f2c849e08465260baf5091a29aa0b25f4fb WatchSource:0}: Error finding container 46b4652eb3fc84d3d146115a78f81f2c849e08465260baf5091a29aa0b25f4fb: Status 404 returned error can't find the container with id 46b4652eb3fc84d3d146115a78f81f2c849e08465260baf5091a29aa0b25f4fb Nov 03 01:56:57.756401 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:57.756354 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-b177c9/ipp-route-b177c9-kserve-7b945c8cfc-gk9vl"] Nov 03 01:56:57.817739 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:57.817713 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-b177c9/ipp-route-b177c9-kserve-7b945c8cfc-gk9vl" event={"ID":"b29aa32a-3985-4f47-906c-e27b88d21617","Type":"ContainerStarted","Data":"d02316ce33689ab54f068cf1f43d47fc7d889b89ecff9fcb769bcfc2f22194f9"} Nov 03 01:56:57.817877 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:57.817762 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-b177c9/ipp-route-b177c9-kserve-7b945c8cfc-gk9vl" event={"ID":"b29aa32a-3985-4f47-906c-e27b88d21617","Type":"ContainerStarted","Data":"46b4652eb3fc84d3d146115a78f81f2c849e08465260baf5091a29aa0b25f4fb"} Nov 03 01:56:57.939622 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:57.939563 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-b177c9/ipp-route-b177c9-kserve-7b945c8cfc-gk9vl" podStartSLOduration=0.939548959 podStartE2EDuration="939.548959ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:56:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:56:57.937246271 +0000 UTC m=+2299.966728566" watchObservedRunningTime="2026-11-03 01:56:57.939548959 +0000 UTC m=+2299.969031270" Nov 03 01:56:58.381037 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:56:58.381008 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-b177c9/ipp-route-b177c9-kserve-7b945c8cfc-gk9vl" Nov 03 01:57:00.193028 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:00.192987 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:57:00.775969 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:00.775934 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-70235a-74f5459759-7hw6j" Nov 03 01:57:01.780913 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:01.780883 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-70235a-6f9d5df8dc-xgzc9" Nov 03 01:57:03.794993 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:03.794961 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f95773-79bdcd49ff-mvz76" Nov 03 01:57:05.084020 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:05.083985 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:57:07.380960 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:07.380923 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-b177c9/ipp-route-b177c9-kserve-7b945c8cfc-gk9vl" Nov 03 01:57:07.397294 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:07.397264 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-b177c9/ipp-route-b177c9-kserve-7b945c8cfc-gk9vl" Nov 03 01:57:07.888582 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:07.888547 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-b177c9/ipp-route-b177c9-kserve-7b945c8cfc-gk9vl" Nov 03 01:57:09.876948 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:09.876907 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:57:14.035079 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:57:14.035042 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 01:57:14.669955 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:14.669920 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:57:19.378872 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:19.378829 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:57:24.041912 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:24.041874 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:57:24.246863 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:24.246823 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:57:29.057048 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:29.057008 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:57:31.950121 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:31.950090 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-zjbzg"] Nov 03 01:57:31.950569 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:31.950291 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-zjbzg" podUID="88142666-5b4b-40db-815e-73c0d8db0993" containerName="istio-proxy" containerID="cri-o://44084a92134019fa7c9d68c0f465ad6ff2a72259c571adc9774c53756938ab2e" gracePeriod=30 Nov 03 01:57:32.289132 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:32.289038 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:57:33.483779 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:33.483746 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:57:34.255141 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:34.255100 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:57:34.463739 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:34.463700 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:57:35.282855 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:35.282825 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-2tzm2"] Nov 03 01:57:35.283279 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:35.283076 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-2tzm2" podUID="f1006e22-dda3-4a64-b5c2-a711362b032d" containerName="istio-proxy" containerID="cri-o://bfae63d3f41e34e4b7044ec53e391b5dc6a26d5b0e7a39ff82029de31d82b064" gracePeriod=30 Nov 03 01:57:35.720075 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:35.720041 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:57:35.934694 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:35.934649 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:57:36.134863 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:36.134825 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:57:37.237142 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:37.237107 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-zjbzg" Nov 03 01:57:37.322377 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:37.322325 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/88142666-5b4b-40db-815e-73c0d8db0993-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88142666-5b4b-40db-815e-73c0d8db0993-istio-podinfo\") pod \"88142666-5b4b-40db-815e-73c0d8db0993\" (UID: \"88142666-5b4b-40db-815e-73c0d8db0993\") " Nov 03 01:57:37.322567 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:37.322386 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88142666-5b4b-40db-815e-73c0d8db0993-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88142666-5b4b-40db-815e-73c0d8db0993-istio-envoy\") pod \"88142666-5b4b-40db-815e-73c0d8db0993\" (UID: \"88142666-5b4b-40db-815e-73c0d8db0993\") " Nov 03 01:57:37.322567 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:37.322474 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88142666-5b4b-40db-815e-73c0d8db0993-kube-api-access-j66qj\" (UniqueName: \"kubernetes.io/projected/88142666-5b4b-40db-815e-73c0d8db0993-kube-api-access-j66qj\") pod \"88142666-5b4b-40db-815e-73c0d8db0993\" (UID: \"88142666-5b4b-40db-815e-73c0d8db0993\") " Nov 03 01:57:37.322567 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:37.322516 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88142666-5b4b-40db-815e-73c0d8db0993-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88142666-5b4b-40db-815e-73c0d8db0993-workload-socket\") pod \"88142666-5b4b-40db-815e-73c0d8db0993\" (UID: \"88142666-5b4b-40db-815e-73c0d8db0993\") " Nov 03 01:57:37.322567 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:37.322544 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88142666-5b4b-40db-815e-73c0d8db0993-istio-token\" (UniqueName: \"kubernetes.io/projected/88142666-5b4b-40db-815e-73c0d8db0993-istio-token\") pod \"88142666-5b4b-40db-815e-73c0d8db0993\" (UID: \"88142666-5b4b-40db-815e-73c0d8db0993\") " Nov 03 01:57:37.322782 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:37.322657 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88142666-5b4b-40db-815e-73c0d8db0993-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88142666-5b4b-40db-815e-73c0d8db0993-credential-socket\") pod \"88142666-5b4b-40db-815e-73c0d8db0993\" (UID: \"88142666-5b4b-40db-815e-73c0d8db0993\") " Nov 03 01:57:37.322782 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:37.322725 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88142666-5b4b-40db-815e-73c0d8db0993-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88142666-5b4b-40db-815e-73c0d8db0993-workload-certs\") pod \"88142666-5b4b-40db-815e-73c0d8db0993\" (UID: \"88142666-5b4b-40db-815e-73c0d8db0993\") " Nov 03 01:57:37.322912 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:37.322895 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88142666-5b4b-40db-815e-73c0d8db0993-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88142666-5b4b-40db-815e-73c0d8db0993-istiod-ca-cert\") pod \"88142666-5b4b-40db-815e-73c0d8db0993\" (UID: \"88142666-5b4b-40db-815e-73c0d8db0993\") " Nov 03 01:57:37.322975 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:37.322939 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88142666-5b4b-40db-815e-73c0d8db0993-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88142666-5b4b-40db-815e-73c0d8db0993-istio-data\") pod \"88142666-5b4b-40db-815e-73c0d8db0993\" (UID: \"88142666-5b4b-40db-815e-73c0d8db0993\") " Nov 03 01:57:37.323495 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:37.323146 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88142666-5b4b-40db-815e-73c0d8db0993-workload-certs" pod "88142666-5b4b-40db-815e-73c0d8db0993" (UID: "88142666-5b4b-40db-815e-73c0d8db0993"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:57:37.323495 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:37.323414 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88142666-5b4b-40db-815e-73c0d8db0993-istio-data" pod "88142666-5b4b-40db-815e-73c0d8db0993" (UID: "88142666-5b4b-40db-815e-73c0d8db0993"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:57:37.323495 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:37.323428 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88142666-5b4b-40db-815e-73c0d8db0993-workload-socket" pod "88142666-5b4b-40db-815e-73c0d8db0993" (UID: "88142666-5b4b-40db-815e-73c0d8db0993"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:57:37.323495 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:37.323486 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88142666-5b4b-40db-815e-73c0d8db0993-credential-socket" pod "88142666-5b4b-40db-815e-73c0d8db0993" (UID: "88142666-5b4b-40db-815e-73c0d8db0993"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:57:37.323867 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:37.323720 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88142666-5b4b-40db-815e-73c0d8db0993-istiod-ca-cert" pod "88142666-5b4b-40db-815e-73c0d8db0993" (UID: "88142666-5b4b-40db-815e-73c0d8db0993"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:57:37.323867 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:37.323791 5437 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88142666-5b4b-40db-815e-73c0d8db0993-credential-socket\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:57:37.323867 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:37.323817 5437 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88142666-5b4b-40db-815e-73c0d8db0993-workload-certs\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:57:37.323867 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:37.323857 5437 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88142666-5b4b-40db-815e-73c0d8db0993-istio-data\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:57:37.324077 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:37.323873 5437 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88142666-5b4b-40db-815e-73c0d8db0993-workload-socket\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:57:37.324993 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:37.324964 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88142666-5b4b-40db-815e-73c0d8db0993-kube-api-access-j66qj" pod "88142666-5b4b-40db-815e-73c0d8db0993" (UID: "88142666-5b4b-40db-815e-73c0d8db0993"). InnerVolumeSpecName "kube-api-access-j66qj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:57:37.325103 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:37.325018 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/88142666-5b4b-40db-815e-73c0d8db0993-istio-podinfo" pod "88142666-5b4b-40db-815e-73c0d8db0993" (UID: "88142666-5b4b-40db-815e-73c0d8db0993"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:57:37.325103 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:37.325082 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88142666-5b4b-40db-815e-73c0d8db0993-istio-token" pod "88142666-5b4b-40db-815e-73c0d8db0993" (UID: "88142666-5b4b-40db-815e-73c0d8db0993"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:57:37.325216 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:37.325139 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88142666-5b4b-40db-815e-73c0d8db0993-istio-envoy" pod "88142666-5b4b-40db-815e-73c0d8db0993" (UID: "88142666-5b4b-40db-815e-73c0d8db0993"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:57:37.426645 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:37.426612 5437 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88142666-5b4b-40db-815e-73c0d8db0993-istiod-ca-cert\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:57:37.426645 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:37.426644 5437 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88142666-5b4b-40db-815e-73c0d8db0993-istio-podinfo\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:57:37.426848 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:37.426659 5437 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88142666-5b4b-40db-815e-73c0d8db0993-istio-envoy\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:57:37.426848 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:37.426672 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j66qj\" (UniqueName: \"kubernetes.io/projected/88142666-5b4b-40db-815e-73c0d8db0993-kube-api-access-j66qj\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:57:37.426848 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:37.426686 5437 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88142666-5b4b-40db-815e-73c0d8db0993-istio-token\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:57:37.605164 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:37.605129 5437 patch_prober.go:29] interesting pod/e2e-worker-w2-9da431-openshift-default-65c799fd77-2tzm2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 01:57:37.605340 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:37.605193 5437 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-2tzm2" podUID="f1006e22-dda3-4a64-b5c2-a711362b032d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 01:57:37.911861 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:37.911773 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:57:38.054812 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:38.054775 5437 generic.go:356] "Generic (PLEG): container finished" podID="88142666-5b4b-40db-815e-73c0d8db0993" containerID="44084a92134019fa7c9d68c0f465ad6ff2a72259c571adc9774c53756938ab2e" exitCode=0 Nov 03 01:57:38.055020 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:38.054824 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-zjbzg" Nov 03 01:57:38.055020 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:38.054859 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-zjbzg" event={"ID":"88142666-5b4b-40db-815e-73c0d8db0993","Type":"ContainerDied","Data":"44084a92134019fa7c9d68c0f465ad6ff2a72259c571adc9774c53756938ab2e"} Nov 03 01:57:38.055020 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:38.054916 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-zjbzg" event={"ID":"88142666-5b4b-40db-815e-73c0d8db0993","Type":"ContainerDied","Data":"3da511ced694af6a9258aebcf517d401bdc715f4fdde532ebcc3a37be975b4ad"} Nov 03 01:57:38.055020 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:38.054970 5437 scope.go:122] "RemoveContainer" containerID="44084a92134019fa7c9d68c0f465ad6ff2a72259c571adc9774c53756938ab2e" Nov 03 01:57:38.069044 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:38.067828 5437 scope.go:122] "RemoveContainer" containerID="44084a92134019fa7c9d68c0f465ad6ff2a72259c571adc9774c53756938ab2e" Nov 03 01:57:38.069044 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:57:38.068645 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44084a92134019fa7c9d68c0f465ad6ff2a72259c571adc9774c53756938ab2e\": container with ID starting with 44084a92134019fa7c9d68c0f465ad6ff2a72259c571adc9774c53756938ab2e not found: ID does not exist" containerID="44084a92134019fa7c9d68c0f465ad6ff2a72259c571adc9774c53756938ab2e" Nov 03 01:57:38.069044 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:38.068678 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44084a92134019fa7c9d68c0f465ad6ff2a72259c571adc9774c53756938ab2e"} err="failed to get container status \"44084a92134019fa7c9d68c0f465ad6ff2a72259c571adc9774c53756938ab2e\": rpc error: code = NotFound desc = could not find container \"44084a92134019fa7c9d68c0f465ad6ff2a72259c571adc9774c53756938ab2e\": container with ID starting with 44084a92134019fa7c9d68c0f465ad6ff2a72259c571adc9774c53756938ab2e not found: ID does not exist" Nov 03 01:57:38.133463 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:38.133424 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:57:38.231103 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:38.231029 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-zjbzg"] Nov 03 01:57:38.238664 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:38.238636 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-zjbzg"] Nov 03 01:57:40.090112 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:40.090077 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88142666-5b4b-40db-815e-73c0d8db0993" path="/var/lib/kubelet/pods/88142666-5b4b-40db-815e-73c0d8db0993/volumes" Nov 03 01:57:40.544127 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:40.544099 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-2tzm2" Nov 03 01:57:40.665122 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:40.665030 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1006e22-dda3-4a64-b5c2-a711362b032d-istio-token\" (UniqueName: \"kubernetes.io/projected/f1006e22-dda3-4a64-b5c2-a711362b032d-istio-token\") pod \"f1006e22-dda3-4a64-b5c2-a711362b032d\" (UID: \"f1006e22-dda3-4a64-b5c2-a711362b032d\") " Nov 03 01:57:40.665122 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:40.665067 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1006e22-dda3-4a64-b5c2-a711362b032d-kube-api-access-dqrgx\" (UniqueName: \"kubernetes.io/projected/f1006e22-dda3-4a64-b5c2-a711362b032d-kube-api-access-dqrgx\") pod \"f1006e22-dda3-4a64-b5c2-a711362b032d\" (UID: \"f1006e22-dda3-4a64-b5c2-a711362b032d\") " Nov 03 01:57:40.665122 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:40.665096 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1006e22-dda3-4a64-b5c2-a711362b032d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1006e22-dda3-4a64-b5c2-a711362b032d-workload-certs\") pod \"f1006e22-dda3-4a64-b5c2-a711362b032d\" (UID: \"f1006e22-dda3-4a64-b5c2-a711362b032d\") " Nov 03 01:57:40.665423 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:40.665157 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1006e22-dda3-4a64-b5c2-a711362b032d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1006e22-dda3-4a64-b5c2-a711362b032d-workload-socket\") pod \"f1006e22-dda3-4a64-b5c2-a711362b032d\" (UID: \"f1006e22-dda3-4a64-b5c2-a711362b032d\") " Nov 03 01:57:40.665423 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:40.665245 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1006e22-dda3-4a64-b5c2-a711362b032d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1006e22-dda3-4a64-b5c2-a711362b032d-istiod-ca-cert\") pod \"f1006e22-dda3-4a64-b5c2-a711362b032d\" (UID: \"f1006e22-dda3-4a64-b5c2-a711362b032d\") " Nov 03 01:57:40.665423 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:40.665391 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1006e22-dda3-4a64-b5c2-a711362b032d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1006e22-dda3-4a64-b5c2-a711362b032d-istio-data\") pod \"f1006e22-dda3-4a64-b5c2-a711362b032d\" (UID: \"f1006e22-dda3-4a64-b5c2-a711362b032d\") " Nov 03 01:57:40.665423 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:40.665420 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1006e22-dda3-4a64-b5c2-a711362b032d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1006e22-dda3-4a64-b5c2-a711362b032d-credential-socket\") pod \"f1006e22-dda3-4a64-b5c2-a711362b032d\" (UID: \"f1006e22-dda3-4a64-b5c2-a711362b032d\") " Nov 03 01:57:40.665648 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:40.665464 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1006e22-dda3-4a64-b5c2-a711362b032d-workload-certs" pod "f1006e22-dda3-4a64-b5c2-a711362b032d" (UID: "f1006e22-dda3-4a64-b5c2-a711362b032d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:57:40.665648 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:40.665487 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f1006e22-dda3-4a64-b5c2-a711362b032d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1006e22-dda3-4a64-b5c2-a711362b032d-istio-podinfo\") pod \"f1006e22-dda3-4a64-b5c2-a711362b032d\" (UID: \"f1006e22-dda3-4a64-b5c2-a711362b032d\") " Nov 03 01:57:40.665648 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:40.665527 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1006e22-dda3-4a64-b5c2-a711362b032d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1006e22-dda3-4a64-b5c2-a711362b032d-istio-envoy\") pod \"f1006e22-dda3-4a64-b5c2-a711362b032d\" (UID: \"f1006e22-dda3-4a64-b5c2-a711362b032d\") " Nov 03 01:57:40.665648 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:40.665579 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1006e22-dda3-4a64-b5c2-a711362b032d-workload-socket" pod "f1006e22-dda3-4a64-b5c2-a711362b032d" (UID: "f1006e22-dda3-4a64-b5c2-a711362b032d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:57:40.665874 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:40.665665 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1006e22-dda3-4a64-b5c2-a711362b032d-istio-data" pod "f1006e22-dda3-4a64-b5c2-a711362b032d" (UID: "f1006e22-dda3-4a64-b5c2-a711362b032d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:57:40.665874 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:40.665697 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1006e22-dda3-4a64-b5c2-a711362b032d-credential-socket" pod "f1006e22-dda3-4a64-b5c2-a711362b032d" (UID: "f1006e22-dda3-4a64-b5c2-a711362b032d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:57:40.665874 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:40.665769 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1006e22-dda3-4a64-b5c2-a711362b032d-istiod-ca-cert" pod "f1006e22-dda3-4a64-b5c2-a711362b032d" (UID: "f1006e22-dda3-4a64-b5c2-a711362b032d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:57:40.666096 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:40.666069 5437 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1006e22-dda3-4a64-b5c2-a711362b032d-istio-data\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:57:40.666096 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:40.666098 5437 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1006e22-dda3-4a64-b5c2-a711362b032d-credential-socket\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:57:40.666291 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:40.666113 5437 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1006e22-dda3-4a64-b5c2-a711362b032d-workload-certs\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:57:40.666291 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:40.666127 5437 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1006e22-dda3-4a64-b5c2-a711362b032d-workload-socket\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:57:40.666291 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:40.666142 5437 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1006e22-dda3-4a64-b5c2-a711362b032d-istiod-ca-cert\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:57:40.667505 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:40.667483 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1006e22-dda3-4a64-b5c2-a711362b032d-kube-api-access-dqrgx" pod "f1006e22-dda3-4a64-b5c2-a711362b032d" (UID: "f1006e22-dda3-4a64-b5c2-a711362b032d"). InnerVolumeSpecName "kube-api-access-dqrgx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:57:40.667724 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:40.667689 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1006e22-dda3-4a64-b5c2-a711362b032d-istio-token" pod "f1006e22-dda3-4a64-b5c2-a711362b032d" (UID: "f1006e22-dda3-4a64-b5c2-a711362b032d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:57:40.667825 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:40.667735 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f1006e22-dda3-4a64-b5c2-a711362b032d-istio-podinfo" pod "f1006e22-dda3-4a64-b5c2-a711362b032d" (UID: "f1006e22-dda3-4a64-b5c2-a711362b032d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:57:40.668026 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:40.668010 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1006e22-dda3-4a64-b5c2-a711362b032d-istio-envoy" pod "f1006e22-dda3-4a64-b5c2-a711362b032d" (UID: "f1006e22-dda3-4a64-b5c2-a711362b032d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:57:40.767137 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:40.767107 5437 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1006e22-dda3-4a64-b5c2-a711362b032d-istio-podinfo\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:57:40.767137 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:40.767134 5437 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1006e22-dda3-4a64-b5c2-a711362b032d-istio-envoy\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:57:40.767137 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:40.767144 5437 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1006e22-dda3-4a64-b5c2-a711362b032d-istio-token\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:57:40.767435 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:40.767155 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqrgx\" (UniqueName: \"kubernetes.io/projected/f1006e22-dda3-4a64-b5c2-a711362b032d-kube-api-access-dqrgx\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:57:40.901277 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:40.901238 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-70235a-545895b546-88g4c"] Nov 03 01:57:40.902112 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:40.902085 5437 cpu_manager.go:414] "containerMap: removing container" podUID="f1006e22-dda3-4a64-b5c2-a711362b032d" containerName="istio-proxy" Nov 03 01:57:40.902112 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:40.902110 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1006e22-dda3-4a64-b5c2-a711362b032d" containerName="istio-proxy" Nov 03 01:57:40.902352 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:40.902163 5437 cpu_manager.go:414] "containerMap: removing container" podUID="88142666-5b4b-40db-815e-73c0d8db0993" containerName="istio-proxy" Nov 03 01:57:40.902352 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:40.902171 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88142666-5b4b-40db-815e-73c0d8db0993" containerName="istio-proxy" Nov 03 01:57:40.902352 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:40.902300 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="88142666-5b4b-40db-815e-73c0d8db0993" containerName="istio-proxy" Nov 03 01:57:40.902352 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:40.902319 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1006e22-dda3-4a64-b5c2-a711362b032d" containerName="istio-proxy" Nov 03 01:57:40.906435 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:40.906414 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-70235a-545895b546-88g4c" Nov 03 01:57:40.959304 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:40.959213 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-70235a-545895b546-88g4c"] Nov 03 01:57:40.960452 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:40.960426 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-70235a-serving-cert\"" Nov 03 01:57:41.037810 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:41.037772 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:57:41.071083 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:41.071046 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51aabea8-7bcc-49d2-bf06-6b8ba825fd33-metrics-certs\") pod \"maas-api-e2e-praxis-a-70235a-545895b546-88g4c\" (UID: \"51aabea8-7bcc-49d2-bf06-6b8ba825fd33\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-70235a-545895b546-88g4c" Nov 03 01:57:41.071245 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:41.071202 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51aabea8-7bcc-49d2-bf06-6b8ba825fd33-maas-api-tls\") pod \"maas-api-e2e-praxis-a-70235a-545895b546-88g4c\" (UID: \"51aabea8-7bcc-49d2-bf06-6b8ba825fd33\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-70235a-545895b546-88g4c" Nov 03 01:57:41.071453 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:41.071425 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51aabea8-7bcc-49d2-bf06-6b8ba825fd33-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-70235a-545895b546-88g4c\" (UID: \"51aabea8-7bcc-49d2-bf06-6b8ba825fd33\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-70235a-545895b546-88g4c" Nov 03 01:57:41.071594 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:41.071569 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hr99\" (UniqueName: \"kubernetes.io/projected/51aabea8-7bcc-49d2-bf06-6b8ba825fd33-kube-api-access-4hr99\") pod \"maas-api-e2e-praxis-a-70235a-545895b546-88g4c\" (UID: \"51aabea8-7bcc-49d2-bf06-6b8ba825fd33\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-70235a-545895b546-88g4c" Nov 03 01:57:41.072559 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:41.072528 5437 generic.go:356] "Generic (PLEG): container finished" podID="f1006e22-dda3-4a64-b5c2-a711362b032d" containerID="bfae63d3f41e34e4b7044ec53e391b5dc6a26d5b0e7a39ff82029de31d82b064" exitCode=0 Nov 03 01:57:41.072667 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:41.072575 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-2tzm2" Nov 03 01:57:41.072667 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:41.072595 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-2tzm2" event={"ID":"f1006e22-dda3-4a64-b5c2-a711362b032d","Type":"ContainerDied","Data":"bfae63d3f41e34e4b7044ec53e391b5dc6a26d5b0e7a39ff82029de31d82b064"} Nov 03 01:57:41.072667 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:41.072636 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-2tzm2" event={"ID":"f1006e22-dda3-4a64-b5c2-a711362b032d","Type":"ContainerDied","Data":"b3f563adacdac3417659eb40675e10de87f637e9ce3591c2bad9f459e4fc0399"} Nov 03 01:57:41.072667 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:41.072660 5437 scope.go:122] "RemoveContainer" containerID="bfae63d3f41e34e4b7044ec53e391b5dc6a26d5b0e7a39ff82029de31d82b064" Nov 03 01:57:41.086001 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:41.085905 5437 scope.go:122] "RemoveContainer" containerID="bfae63d3f41e34e4b7044ec53e391b5dc6a26d5b0e7a39ff82029de31d82b064" Nov 03 01:57:41.086275 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:57:41.086249 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfae63d3f41e34e4b7044ec53e391b5dc6a26d5b0e7a39ff82029de31d82b064\": container with ID starting with bfae63d3f41e34e4b7044ec53e391b5dc6a26d5b0e7a39ff82029de31d82b064 not found: ID does not exist" containerID="bfae63d3f41e34e4b7044ec53e391b5dc6a26d5b0e7a39ff82029de31d82b064" Nov 03 01:57:41.086334 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:41.086289 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfae63d3f41e34e4b7044ec53e391b5dc6a26d5b0e7a39ff82029de31d82b064"} err="failed to get container status \"bfae63d3f41e34e4b7044ec53e391b5dc6a26d5b0e7a39ff82029de31d82b064\": rpc error: code = NotFound desc = could not find container \"bfae63d3f41e34e4b7044ec53e391b5dc6a26d5b0e7a39ff82029de31d82b064\": container with ID starting with bfae63d3f41e34e4b7044ec53e391b5dc6a26d5b0e7a39ff82029de31d82b064 not found: ID does not exist" Nov 03 01:57:41.172297 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:41.172246 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51aabea8-7bcc-49d2-bf06-6b8ba825fd33-maas-api-tls\") pod \"maas-api-e2e-praxis-a-70235a-545895b546-88g4c\" (UID: \"51aabea8-7bcc-49d2-bf06-6b8ba825fd33\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-70235a-545895b546-88g4c" Nov 03 01:57:41.172802 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:41.172388 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51aabea8-7bcc-49d2-bf06-6b8ba825fd33-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-70235a-545895b546-88g4c\" (UID: \"51aabea8-7bcc-49d2-bf06-6b8ba825fd33\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-70235a-545895b546-88g4c" Nov 03 01:57:41.172802 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:41.172451 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hr99\" (UniqueName: \"kubernetes.io/projected/51aabea8-7bcc-49d2-bf06-6b8ba825fd33-kube-api-access-4hr99\") pod \"maas-api-e2e-praxis-a-70235a-545895b546-88g4c\" (UID: \"51aabea8-7bcc-49d2-bf06-6b8ba825fd33\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-70235a-545895b546-88g4c" Nov 03 01:57:41.172802 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:57:41.172456 5437 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-70235a-serving-cert: secret "maas-api-e2e-praxis-a-70235a-serving-cert" not found Nov 03 01:57:41.172802 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:41.172490 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51aabea8-7bcc-49d2-bf06-6b8ba825fd33-metrics-certs\") pod \"maas-api-e2e-praxis-a-70235a-545895b546-88g4c\" (UID: \"51aabea8-7bcc-49d2-bf06-6b8ba825fd33\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-70235a-545895b546-88g4c" Nov 03 01:57:41.172802 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:57:41.172551 5437 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51aabea8-7bcc-49d2-bf06-6b8ba825fd33-maas-api-tls podName:51aabea8-7bcc-49d2-bf06-6b8ba825fd33 nodeName:}" failed. No retries permitted until 2026-11-03 01:57:41.672528071 +0000 UTC m=+2343.702010360 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/51aabea8-7bcc-49d2-bf06-6b8ba825fd33-maas-api-tls") pod "maas-api-e2e-praxis-a-70235a-545895b546-88g4c" (UID: "51aabea8-7bcc-49d2-bf06-6b8ba825fd33") : secret "maas-api-e2e-praxis-a-70235a-serving-cert" not found Nov 03 01:57:41.173198 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:41.173177 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51aabea8-7bcc-49d2-bf06-6b8ba825fd33-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-70235a-545895b546-88g4c\" (UID: \"51aabea8-7bcc-49d2-bf06-6b8ba825fd33\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-70235a-545895b546-88g4c" Nov 03 01:57:41.175051 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:41.175029 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51aabea8-7bcc-49d2-bf06-6b8ba825fd33-metrics-certs\") pod \"maas-api-e2e-praxis-a-70235a-545895b546-88g4c\" (UID: \"51aabea8-7bcc-49d2-bf06-6b8ba825fd33\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-70235a-545895b546-88g4c" Nov 03 01:57:41.237914 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:41.237841 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hr99\" (UniqueName: \"kubernetes.io/projected/51aabea8-7bcc-49d2-bf06-6b8ba825fd33-kube-api-access-4hr99\") pod \"maas-api-e2e-praxis-a-70235a-545895b546-88g4c\" (UID: \"51aabea8-7bcc-49d2-bf06-6b8ba825fd33\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-70235a-545895b546-88g4c" Nov 03 01:57:41.242689 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:41.242659 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-2tzm2"] Nov 03 01:57:41.246754 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:41.246732 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-2tzm2"] Nov 03 01:57:41.287096 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:41.287060 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:57:41.384834 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:41.384804 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-a00717-76799c76dc-7jl9f"] Nov 03 01:57:41.390240 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:41.390217 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a00717-76799c76dc-7jl9f" Nov 03 01:57:41.443762 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:41.443726 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-a00717-76799c76dc-7jl9f"] Nov 03 01:57:41.445052 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:41.444679 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-a00717-dockercfg-zwzph\"" Nov 03 01:57:41.445052 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:41.444781 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-a00717\"" Nov 03 01:57:41.477733 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:41.477696 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a255ffd8-6415-41b7-af49-0e6522ff1133-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-a00717-76799c76dc-7jl9f\" (UID: \"a255ffd8-6415-41b7-af49-0e6522ff1133\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a00717-76799c76dc-7jl9f" Nov 03 01:57:41.478049 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:41.478016 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49287\" (UniqueName: \"kubernetes.io/projected/a255ffd8-6415-41b7-af49-0e6522ff1133-kube-api-access-49287\") pod \"payload-pre-processing-e2e-praxis-b-a00717-76799c76dc-7jl9f\" (UID: \"a255ffd8-6415-41b7-af49-0e6522ff1133\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a00717-76799c76dc-7jl9f" Nov 03 01:57:41.496108 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:41.495977 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:57:41.533898 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:41.533867 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-a00717-84ffc588b5-k8cfr"] Nov 03 01:57:41.537888 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:41.537867 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-a00717-84ffc588b5-k8cfr" Nov 03 01:57:41.579744 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:41.579705 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49287\" (UniqueName: \"kubernetes.io/projected/a255ffd8-6415-41b7-af49-0e6522ff1133-kube-api-access-49287\") pod \"payload-pre-processing-e2e-praxis-b-a00717-76799c76dc-7jl9f\" (UID: \"a255ffd8-6415-41b7-af49-0e6522ff1133\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a00717-76799c76dc-7jl9f" Nov 03 01:57:41.579915 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:41.579828 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a255ffd8-6415-41b7-af49-0e6522ff1133-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-a00717-76799c76dc-7jl9f\" (UID: \"a255ffd8-6415-41b7-af49-0e6522ff1133\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a00717-76799c76dc-7jl9f" Nov 03 01:57:41.580550 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:41.580530 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a255ffd8-6415-41b7-af49-0e6522ff1133-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-a00717-76799c76dc-7jl9f\" (UID: \"a255ffd8-6415-41b7-af49-0e6522ff1133\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a00717-76799c76dc-7jl9f" Nov 03 01:57:41.592326 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:41.592296 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-a00717-84ffc588b5-k8cfr"] Nov 03 01:57:41.643072 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:41.643038 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49287\" (UniqueName: \"kubernetes.io/projected/a255ffd8-6415-41b7-af49-0e6522ff1133-kube-api-access-49287\") pod \"payload-pre-processing-e2e-praxis-b-a00717-76799c76dc-7jl9f\" (UID: \"a255ffd8-6415-41b7-af49-0e6522ff1133\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a00717-76799c76dc-7jl9f" Nov 03 01:57:41.680793 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:41.680764 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/775139d4-2b62-484e-8d22-ae25e9e2d4d5-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-a00717-84ffc588b5-k8cfr\" (UID: \"775139d4-2b62-484e-8d22-ae25e9e2d4d5\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-a00717-84ffc588b5-k8cfr" Nov 03 01:57:41.680967 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:41.680947 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5bhw\" (UniqueName: \"kubernetes.io/projected/775139d4-2b62-484e-8d22-ae25e9e2d4d5-kube-api-access-j5bhw\") pod \"payload-processing-e2e-praxis-b-a00717-84ffc588b5-k8cfr\" (UID: \"775139d4-2b62-484e-8d22-ae25e9e2d4d5\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-a00717-84ffc588b5-k8cfr" Nov 03 01:57:41.681025 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:41.681013 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51aabea8-7bcc-49d2-bf06-6b8ba825fd33-maas-api-tls\") pod \"maas-api-e2e-praxis-a-70235a-545895b546-88g4c\" (UID: \"51aabea8-7bcc-49d2-bf06-6b8ba825fd33\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-70235a-545895b546-88g4c" Nov 03 01:57:41.683560 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:41.683538 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51aabea8-7bcc-49d2-bf06-6b8ba825fd33-maas-api-tls\") pod \"maas-api-e2e-praxis-a-70235a-545895b546-88g4c\" (UID: \"51aabea8-7bcc-49d2-bf06-6b8ba825fd33\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-70235a-545895b546-88g4c" Nov 03 01:57:41.704412 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:41.704357 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a00717-76799c76dc-7jl9f" Nov 03 01:57:41.782493 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:41.782460 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5bhw\" (UniqueName: \"kubernetes.io/projected/775139d4-2b62-484e-8d22-ae25e9e2d4d5-kube-api-access-j5bhw\") pod \"payload-processing-e2e-praxis-b-a00717-84ffc588b5-k8cfr\" (UID: \"775139d4-2b62-484e-8d22-ae25e9e2d4d5\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-a00717-84ffc588b5-k8cfr" Nov 03 01:57:41.782652 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:41.782621 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/775139d4-2b62-484e-8d22-ae25e9e2d4d5-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-a00717-84ffc588b5-k8cfr\" (UID: \"775139d4-2b62-484e-8d22-ae25e9e2d4d5\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-a00717-84ffc588b5-k8cfr" Nov 03 01:57:41.783433 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:41.783408 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/775139d4-2b62-484e-8d22-ae25e9e2d4d5-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-a00717-84ffc588b5-k8cfr\" (UID: \"775139d4-2b62-484e-8d22-ae25e9e2d4d5\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-a00717-84ffc588b5-k8cfr" Nov 03 01:57:41.822561 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:41.822526 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-70235a-545895b546-88g4c" Nov 03 01:57:41.845295 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:41.845272 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5bhw\" (UniqueName: \"kubernetes.io/projected/775139d4-2b62-484e-8d22-ae25e9e2d4d5-kube-api-access-j5bhw\") pod \"payload-processing-e2e-praxis-b-a00717-84ffc588b5-k8cfr\" (UID: \"775139d4-2b62-484e-8d22-ae25e9e2d4d5\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-a00717-84ffc588b5-k8cfr" Nov 03 01:57:41.857794 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:41.857754 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-a00717-84ffc588b5-k8cfr" Nov 03 01:57:42.079659 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:57:42.079618 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda255ffd8_6415_41b7_af49_0e6522ff1133.slice/crio-8806c3255933b86832a6761373c368f9379618b05c2e43af061761c0fb6c76d8 WatchSource:0}: Error finding container 8806c3255933b86832a6761373c368f9379618b05c2e43af061761c0fb6c76d8: Status 404 returned error can't find the container with id 8806c3255933b86832a6761373c368f9379618b05c2e43af061761c0fb6c76d8 Nov 03 01:57:42.091568 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:42.091542 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1006e22-dda3-4a64-b5c2-a711362b032d" path="/var/lib/kubelet/pods/f1006e22-dda3-4a64-b5c2-a711362b032d/volumes" Nov 03 01:57:42.091859 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:42.091845 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-a00717-76799c76dc-7jl9f"] Nov 03 01:57:42.199089 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:57:42.199062 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51aabea8_7bcc_49d2_bf06_6b8ba825fd33.slice/crio-9dddce151ed19d54a4d28299f6ff9474f0f54f2c7568f9db5eb2b7a05c7effd4 WatchSource:0}: Error finding container 9dddce151ed19d54a4d28299f6ff9474f0f54f2c7568f9db5eb2b7a05c7effd4: Status 404 returned error can't find the container with id 9dddce151ed19d54a4d28299f6ff9474f0f54f2c7568f9db5eb2b7a05c7effd4 Nov 03 01:57:42.200644 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:42.200623 5437 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 01:57:42.202390 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:42.202359 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-70235a-545895b546-88g4c"] Nov 03 01:57:42.245786 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:57:42.245761 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod775139d4_2b62_484e_8d22_ae25e9e2d4d5.slice/crio-74d95f7402fc788181645f8a20a946c03afaf943eed3abc628a27be41a47ee99 WatchSource:0}: Error finding container 74d95f7402fc788181645f8a20a946c03afaf943eed3abc628a27be41a47ee99: Status 404 returned error can't find the container with id 74d95f7402fc788181645f8a20a946c03afaf943eed3abc628a27be41a47ee99 Nov 03 01:57:42.249553 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:42.249531 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-a00717-84ffc588b5-k8cfr"] Nov 03 01:57:43.091624 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:43.091591 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-70235a-545895b546-88g4c" event={"ID":"51aabea8-7bcc-49d2-bf06-6b8ba825fd33","Type":"ContainerStarted","Data":"9dddce151ed19d54a4d28299f6ff9474f0f54f2c7568f9db5eb2b7a05c7effd4"} Nov 03 01:57:43.093422 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:43.093395 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a00717-76799c76dc-7jl9f" event={"ID":"a255ffd8-6415-41b7-af49-0e6522ff1133","Type":"ContainerStarted","Data":"c5c37b40b4373fe19616ff677acb2e3fcaff97f899fa6d1901994f7e685db86d"} Nov 03 01:57:43.093548 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:43.093449 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a00717-76799c76dc-7jl9f" event={"ID":"a255ffd8-6415-41b7-af49-0e6522ff1133","Type":"ContainerStarted","Data":"8806c3255933b86832a6761373c368f9379618b05c2e43af061761c0fb6c76d8"} Nov 03 01:57:43.093657 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:43.093634 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a00717-76799c76dc-7jl9f" Nov 03 01:57:43.095107 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:43.095079 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-a00717-84ffc588b5-k8cfr" event={"ID":"775139d4-2b62-484e-8d22-ae25e9e2d4d5","Type":"ContainerStarted","Data":"12bdbdf61526e81cec961a8dfd57a6d64981c917ca4d8d3ec4e667cf374648c3"} Nov 03 01:57:43.095204 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:43.095120 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-a00717-84ffc588b5-k8cfr" event={"ID":"775139d4-2b62-484e-8d22-ae25e9e2d4d5","Type":"ContainerStarted","Data":"74d95f7402fc788181645f8a20a946c03afaf943eed3abc628a27be41a47ee99"} Nov 03 01:57:43.095204 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:43.095151 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-a00717-84ffc588b5-k8cfr" Nov 03 01:57:43.215379 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:43.215306 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a00717-76799c76dc-7jl9f" podStartSLOduration=2.21529226 podStartE2EDuration="2.21529226s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:57:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:57:43.212781421 +0000 UTC m=+2345.242263717" watchObservedRunningTime="2026-11-03 01:57:43.21529226 +0000 UTC m=+2345.244774554" Nov 03 01:57:43.333937 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:43.333792 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-a00717-84ffc588b5-k8cfr" podStartSLOduration=2.333776231 podStartE2EDuration="2.333776231s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:57:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:57:43.332064775 +0000 UTC m=+2345.361547098" watchObservedRunningTime="2026-11-03 01:57:43.333776231 +0000 UTC m=+2345.363258526" Nov 03 01:57:44.040388 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:57:44.040324 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 01:57:44.102496 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:44.102455 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-70235a-545895b546-88g4c" event={"ID":"51aabea8-7bcc-49d2-bf06-6b8ba825fd33","Type":"ContainerStarted","Data":"75df0b4c7c96f5fb902de6e6680c5c34496f7e1ad0a4a9a49ac123b0ed7c4a69"} Nov 03 01:57:44.102675 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:44.102520 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-70235a-545895b546-88g4c" Nov 03 01:57:44.257655 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:44.257602 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-70235a-545895b546-88g4c" podStartSLOduration=3.38018337 podStartE2EDuration="4.257587798s" totalImagesPullingTime="877.404428ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:57:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:57:44.255820085 +0000 UTC m=+2346.285302381" watchObservedRunningTime="2026-11-03 01:57:44.257587798 +0000 UTC m=+2346.287070092" Nov 03 01:57:46.229562 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:46.229524 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:57:46.437912 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:46.437876 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:57:46.642782 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:46.642737 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:57:49.093817 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:49.093772 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:57:50.113451 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:50.113422 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-70235a-545895b546-88g4c" Nov 03 01:57:51.191806 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:51.191768 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:57:53.666509 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:53.666471 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:57:54.106057 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:54.106026 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-a00717-84ffc588b5-k8cfr" Nov 03 01:57:54.106236 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:54.106102 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a00717-76799c76dc-7jl9f" Nov 03 01:57:56.172749 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:57:56.172708 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:58:02.496776 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:58:02.496742 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:58:07.270755 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:58:07.270719 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:58:12.080845 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:58:12.080803 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:58:14.036116 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:58:14.036074 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 01:58:16.918535 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:58:16.918474 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:58:21.611237 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:58:21.611200 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:58:26.323319 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:58:26.323281 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:58:31.067844 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:58:31.067793 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:58:35.990221 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:58:35.990178 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:58:42.383200 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:58:42.383146 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:58:44.034909 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:58:44.034868 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 01:58:47.130091 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:58:47.130058 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:58:47.332914 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:58:47.332876 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:58:47.538276 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:58:47.538177 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:58:48.311805 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:58:48.311766 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:58:49.452762 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:58:49.452723 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:58:52.259745 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:58:52.259705 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:58:53.336757 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:58:53.336719 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:58:54.069636 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:58:54.069584 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:58:57.064015 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:58:57.063973 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:58:58.507502 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:58:58.507466 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:58:59.204844 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:58:59.204808 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:58:59.671174 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:58:59.671141 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:59:00.108796 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:00.108765 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-e0907d-openshift-default-59786747c7-479hs"] Nov 03 01:59:00.114159 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:00.114131 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-e0907d-openshift-default-59786747c7-479hs" Nov 03 01:59:00.168731 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:00.168691 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-e0907d-openshift-default-dockercfg-ttrgd\"" Nov 03 01:59:00.222089 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:00.222043 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ce8bfdb-a81b-4641-996a-c5fe222b1e8c-istio-podinfo\") pod \"e2e-praxis-b-e0907d-openshift-default-59786747c7-479hs\" (UID: \"1ce8bfdb-a81b-4641-996a-c5fe222b1e8c\") " pod="openshift-ingress/e2e-praxis-b-e0907d-openshift-default-59786747c7-479hs" Nov 03 01:59:00.222281 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:00.222171 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ce8bfdb-a81b-4641-996a-c5fe222b1e8c-workload-socket\") pod \"e2e-praxis-b-e0907d-openshift-default-59786747c7-479hs\" (UID: \"1ce8bfdb-a81b-4641-996a-c5fe222b1e8c\") " pod="openshift-ingress/e2e-praxis-b-e0907d-openshift-default-59786747c7-479hs" Nov 03 01:59:00.222337 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:00.222293 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ce8bfdb-a81b-4641-996a-c5fe222b1e8c-istiod-ca-cert\") pod \"e2e-praxis-b-e0907d-openshift-default-59786747c7-479hs\" (UID: \"1ce8bfdb-a81b-4641-996a-c5fe222b1e8c\") " pod="openshift-ingress/e2e-praxis-b-e0907d-openshift-default-59786747c7-479hs" Nov 03 01:59:00.222432 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:00.222352 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ce8bfdb-a81b-4641-996a-c5fe222b1e8c-istio-envoy\") pod \"e2e-praxis-b-e0907d-openshift-default-59786747c7-479hs\" (UID: \"1ce8bfdb-a81b-4641-996a-c5fe222b1e8c\") " pod="openshift-ingress/e2e-praxis-b-e0907d-openshift-default-59786747c7-479hs" Nov 03 01:59:00.222552 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:00.222523 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ce8bfdb-a81b-4641-996a-c5fe222b1e8c-workload-certs\") pod \"e2e-praxis-b-e0907d-openshift-default-59786747c7-479hs\" (UID: \"1ce8bfdb-a81b-4641-996a-c5fe222b1e8c\") " pod="openshift-ingress/e2e-praxis-b-e0907d-openshift-default-59786747c7-479hs" Nov 03 01:59:00.222717 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:00.222586 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ce8bfdb-a81b-4641-996a-c5fe222b1e8c-credential-socket\") pod \"e2e-praxis-b-e0907d-openshift-default-59786747c7-479hs\" (UID: \"1ce8bfdb-a81b-4641-996a-c5fe222b1e8c\") " pod="openshift-ingress/e2e-praxis-b-e0907d-openshift-default-59786747c7-479hs" Nov 03 01:59:00.222717 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:00.222627 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ce8bfdb-a81b-4641-996a-c5fe222b1e8c-istio-data\") pod \"e2e-praxis-b-e0907d-openshift-default-59786747c7-479hs\" (UID: \"1ce8bfdb-a81b-4641-996a-c5fe222b1e8c\") " pod="openshift-ingress/e2e-praxis-b-e0907d-openshift-default-59786747c7-479hs" Nov 03 01:59:00.222717 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:00.222652 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z24pd\" (UniqueName: \"kubernetes.io/projected/1ce8bfdb-a81b-4641-996a-c5fe222b1e8c-kube-api-access-z24pd\") pod \"e2e-praxis-b-e0907d-openshift-default-59786747c7-479hs\" (UID: \"1ce8bfdb-a81b-4641-996a-c5fe222b1e8c\") " pod="openshift-ingress/e2e-praxis-b-e0907d-openshift-default-59786747c7-479hs" Nov 03 01:59:00.222717 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:00.222677 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ce8bfdb-a81b-4641-996a-c5fe222b1e8c-istio-token\") pod \"e2e-praxis-b-e0907d-openshift-default-59786747c7-479hs\" (UID: \"1ce8bfdb-a81b-4641-996a-c5fe222b1e8c\") " pod="openshift-ingress/e2e-praxis-b-e0907d-openshift-default-59786747c7-479hs" Nov 03 01:59:00.223593 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:00.223572 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-e0907d-openshift-default-59786747c7-479hs"] Nov 03 01:59:00.253694 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:00.253653 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:59:00.324248 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:00.324201 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ce8bfdb-a81b-4641-996a-c5fe222b1e8c-istio-podinfo\") pod \"e2e-praxis-b-e0907d-openshift-default-59786747c7-479hs\" (UID: \"1ce8bfdb-a81b-4641-996a-c5fe222b1e8c\") " pod="openshift-ingress/e2e-praxis-b-e0907d-openshift-default-59786747c7-479hs" Nov 03 01:59:00.324477 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:00.324265 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ce8bfdb-a81b-4641-996a-c5fe222b1e8c-workload-socket\") pod \"e2e-praxis-b-e0907d-openshift-default-59786747c7-479hs\" (UID: \"1ce8bfdb-a81b-4641-996a-c5fe222b1e8c\") " pod="openshift-ingress/e2e-praxis-b-e0907d-openshift-default-59786747c7-479hs" Nov 03 01:59:00.324477 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:00.324321 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ce8bfdb-a81b-4641-996a-c5fe222b1e8c-istiod-ca-cert\") pod \"e2e-praxis-b-e0907d-openshift-default-59786747c7-479hs\" (UID: \"1ce8bfdb-a81b-4641-996a-c5fe222b1e8c\") " pod="openshift-ingress/e2e-praxis-b-e0907d-openshift-default-59786747c7-479hs" Nov 03 01:59:00.324477 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:00.324422 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ce8bfdb-a81b-4641-996a-c5fe222b1e8c-istio-envoy\") pod \"e2e-praxis-b-e0907d-openshift-default-59786747c7-479hs\" (UID: \"1ce8bfdb-a81b-4641-996a-c5fe222b1e8c\") " pod="openshift-ingress/e2e-praxis-b-e0907d-openshift-default-59786747c7-479hs" Nov 03 01:59:00.324624 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:00.324501 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ce8bfdb-a81b-4641-996a-c5fe222b1e8c-workload-certs\") pod \"e2e-praxis-b-e0907d-openshift-default-59786747c7-479hs\" (UID: \"1ce8bfdb-a81b-4641-996a-c5fe222b1e8c\") " pod="openshift-ingress/e2e-praxis-b-e0907d-openshift-default-59786747c7-479hs" Nov 03 01:59:00.324624 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:00.324537 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ce8bfdb-a81b-4641-996a-c5fe222b1e8c-credential-socket\") pod \"e2e-praxis-b-e0907d-openshift-default-59786747c7-479hs\" (UID: \"1ce8bfdb-a81b-4641-996a-c5fe222b1e8c\") " pod="openshift-ingress/e2e-praxis-b-e0907d-openshift-default-59786747c7-479hs" Nov 03 01:59:00.324624 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:00.324581 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ce8bfdb-a81b-4641-996a-c5fe222b1e8c-istio-data\") pod \"e2e-praxis-b-e0907d-openshift-default-59786747c7-479hs\" (UID: \"1ce8bfdb-a81b-4641-996a-c5fe222b1e8c\") " pod="openshift-ingress/e2e-praxis-b-e0907d-openshift-default-59786747c7-479hs" Nov 03 01:59:00.324776 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:00.324641 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z24pd\" (UniqueName: \"kubernetes.io/projected/1ce8bfdb-a81b-4641-996a-c5fe222b1e8c-kube-api-access-z24pd\") pod \"e2e-praxis-b-e0907d-openshift-default-59786747c7-479hs\" (UID: \"1ce8bfdb-a81b-4641-996a-c5fe222b1e8c\") " pod="openshift-ingress/e2e-praxis-b-e0907d-openshift-default-59786747c7-479hs" Nov 03 01:59:00.324776 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:00.324674 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ce8bfdb-a81b-4641-996a-c5fe222b1e8c-istio-token\") pod \"e2e-praxis-b-e0907d-openshift-default-59786747c7-479hs\" (UID: \"1ce8bfdb-a81b-4641-996a-c5fe222b1e8c\") " pod="openshift-ingress/e2e-praxis-b-e0907d-openshift-default-59786747c7-479hs" Nov 03 01:59:00.324895 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:00.324863 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ce8bfdb-a81b-4641-996a-c5fe222b1e8c-workload-socket\") pod \"e2e-praxis-b-e0907d-openshift-default-59786747c7-479hs\" (UID: \"1ce8bfdb-a81b-4641-996a-c5fe222b1e8c\") " pod="openshift-ingress/e2e-praxis-b-e0907d-openshift-default-59786747c7-479hs" Nov 03 01:59:00.324960 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:00.324911 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ce8bfdb-a81b-4641-996a-c5fe222b1e8c-workload-certs\") pod \"e2e-praxis-b-e0907d-openshift-default-59786747c7-479hs\" (UID: \"1ce8bfdb-a81b-4641-996a-c5fe222b1e8c\") " pod="openshift-ingress/e2e-praxis-b-e0907d-openshift-default-59786747c7-479hs" Nov 03 01:59:00.325017 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:00.324977 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ce8bfdb-a81b-4641-996a-c5fe222b1e8c-credential-socket\") pod \"e2e-praxis-b-e0907d-openshift-default-59786747c7-479hs\" (UID: \"1ce8bfdb-a81b-4641-996a-c5fe222b1e8c\") " pod="openshift-ingress/e2e-praxis-b-e0907d-openshift-default-59786747c7-479hs" Nov 03 01:59:00.325168 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:00.325125 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ce8bfdb-a81b-4641-996a-c5fe222b1e8c-istio-data\") pod \"e2e-praxis-b-e0907d-openshift-default-59786747c7-479hs\" (UID: \"1ce8bfdb-a81b-4641-996a-c5fe222b1e8c\") " pod="openshift-ingress/e2e-praxis-b-e0907d-openshift-default-59786747c7-479hs" Nov 03 01:59:00.325243 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:00.325202 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ce8bfdb-a81b-4641-996a-c5fe222b1e8c-istiod-ca-cert\") pod \"e2e-praxis-b-e0907d-openshift-default-59786747c7-479hs\" (UID: \"1ce8bfdb-a81b-4641-996a-c5fe222b1e8c\") " pod="openshift-ingress/e2e-praxis-b-e0907d-openshift-default-59786747c7-479hs" Nov 03 01:59:00.327261 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:00.327239 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ce8bfdb-a81b-4641-996a-c5fe222b1e8c-istio-envoy\") pod \"e2e-praxis-b-e0907d-openshift-default-59786747c7-479hs\" (UID: \"1ce8bfdb-a81b-4641-996a-c5fe222b1e8c\") " pod="openshift-ingress/e2e-praxis-b-e0907d-openshift-default-59786747c7-479hs" Nov 03 01:59:00.327615 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:00.327595 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ce8bfdb-a81b-4641-996a-c5fe222b1e8c-istio-podinfo\") pod \"e2e-praxis-b-e0907d-openshift-default-59786747c7-479hs\" (UID: \"1ce8bfdb-a81b-4641-996a-c5fe222b1e8c\") " pod="openshift-ingress/e2e-praxis-b-e0907d-openshift-default-59786747c7-479hs" Nov 03 01:59:00.388267 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:00.388167 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ce8bfdb-a81b-4641-996a-c5fe222b1e8c-istio-token\") pod \"e2e-praxis-b-e0907d-openshift-default-59786747c7-479hs\" (UID: \"1ce8bfdb-a81b-4641-996a-c5fe222b1e8c\") " pod="openshift-ingress/e2e-praxis-b-e0907d-openshift-default-59786747c7-479hs" Nov 03 01:59:00.388267 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:00.388184 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z24pd\" (UniqueName: \"kubernetes.io/projected/1ce8bfdb-a81b-4641-996a-c5fe222b1e8c-kube-api-access-z24pd\") pod \"e2e-praxis-b-e0907d-openshift-default-59786747c7-479hs\" (UID: \"1ce8bfdb-a81b-4641-996a-c5fe222b1e8c\") " pod="openshift-ingress/e2e-praxis-b-e0907d-openshift-default-59786747c7-479hs" Nov 03 01:59:00.433482 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:00.433447 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-e0907d-openshift-default-59786747c7-479hs" Nov 03 01:59:00.685290 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:00.685196 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:59:00.894064 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:59:00.894038 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ce8bfdb_a81b_4641_996a_c5fe222b1e8c.slice/crio-5fd92ccbc98b7db30084da5597e42c4fedcfe0076e8d61823c8f982ec352a3b4 WatchSource:0}: Error finding container 5fd92ccbc98b7db30084da5597e42c4fedcfe0076e8d61823c8f982ec352a3b4: Status 404 returned error can't find the container with id 5fd92ccbc98b7db30084da5597e42c4fedcfe0076e8d61823c8f982ec352a3b4 Nov 03 01:59:00.897058 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:00.897024 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 01:59:00.897161 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:00.897096 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-e0907d-openshift-default-59786747c7-479hs"] Nov 03 01:59:00.897161 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:00.897099 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 01:59:00.897233 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:00.897189 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 01:59:01.291948 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:01.291861 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:59:01.490138 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:01.490103 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:59:01.582194 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:01.582158 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-e0907d-openshift-default-59786747c7-479hs" event={"ID":"1ce8bfdb-a81b-4641-996a-c5fe222b1e8c","Type":"ContainerStarted","Data":"d46029c563cc8d26b81544758389e98c18dbc749005e5d7ec5a52db5cd033d08"} Nov 03 01:59:01.582418 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:01.582213 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-e0907d-openshift-default-59786747c7-479hs" event={"ID":"1ce8bfdb-a81b-4641-996a-c5fe222b1e8c","Type":"ContainerStarted","Data":"5fd92ccbc98b7db30084da5597e42c4fedcfe0076e8d61823c8f982ec352a3b4"} Nov 03 01:59:01.706938 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:01.706888 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-e0907d-openshift-default-59786747c7-479hs" podStartSLOduration=1.706874412 podStartE2EDuration="1.706874412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:59:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:59:01.705966878 +0000 UTC m=+2423.735449173" watchObservedRunningTime="2026-11-03 01:59:01.706874412 +0000 UTC m=+2423.736356707" Nov 03 01:59:02.205194 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:02.205162 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:59:02.433913 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:02.433878 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-e0907d-openshift-default-59786747c7-479hs" Nov 03 01:59:02.440146 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:02.440119 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-e0907d-openshift-default-59786747c7-479hs" Nov 03 01:59:02.588478 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:02.588443 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-e0907d-openshift-default-59786747c7-479hs" Nov 03 01:59:02.589576 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:02.589549 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-e0907d-openshift-default-59786747c7-479hs" Nov 03 01:59:04.458289 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:04.458248 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:59:04.887784 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:04.887745 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:59:05.188649 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:05.188553 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:59:05.385482 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:05.385441 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f95773-79bdcd49ff-mvz76"] Nov 03 01:59:05.410309 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:05.410273 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:59:05.801827 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:05.801788 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:59:06.062953 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:06.062856 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:59:06.270736 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:06.270700 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:59:07.611678 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:07.611643 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:59:09.162081 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:09.162051 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:59:09.425550 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:09.425440 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:59:09.665355 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:09.665319 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-e0907d-74b7bd8f9f-pdk27"] Nov 03 01:59:09.669638 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:09.669618 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-e0907d-74b7bd8f9f-pdk27" Nov 03 01:59:09.724198 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:09.724158 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-e0907d-74b7bd8f9f-pdk27"] Nov 03 01:59:09.748595 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:09.748545 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55jbp\" (UniqueName: \"kubernetes.io/projected/653ed959-fdf9-4ad7-90be-0f58348c80a3-kube-api-access-55jbp\") pod \"payload-processing-e2e-praxis-b-e0907d-74b7bd8f9f-pdk27\" (UID: \"653ed959-fdf9-4ad7-90be-0f58348c80a3\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-e0907d-74b7bd8f9f-pdk27" Nov 03 01:59:09.748812 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:09.748757 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/653ed959-fdf9-4ad7-90be-0f58348c80a3-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-e0907d-74b7bd8f9f-pdk27\" (UID: \"653ed959-fdf9-4ad7-90be-0f58348c80a3\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-e0907d-74b7bd8f9f-pdk27" Nov 03 01:59:09.771962 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:09.771930 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-e0907d\"" Nov 03 01:59:09.772122 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:09.772018 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-e0907d-dockercfg-fjqjs\"" Nov 03 01:59:09.850452 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:09.850415 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55jbp\" (UniqueName: \"kubernetes.io/projected/653ed959-fdf9-4ad7-90be-0f58348c80a3-kube-api-access-55jbp\") pod \"payload-processing-e2e-praxis-b-e0907d-74b7bd8f9f-pdk27\" (UID: \"653ed959-fdf9-4ad7-90be-0f58348c80a3\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-e0907d-74b7bd8f9f-pdk27" Nov 03 01:59:09.850662 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:09.850516 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/653ed959-fdf9-4ad7-90be-0f58348c80a3-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-e0907d-74b7bd8f9f-pdk27\" (UID: \"653ed959-fdf9-4ad7-90be-0f58348c80a3\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-e0907d-74b7bd8f9f-pdk27" Nov 03 01:59:09.851297 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:09.851272 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/653ed959-fdf9-4ad7-90be-0f58348c80a3-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-e0907d-74b7bd8f9f-pdk27\" (UID: \"653ed959-fdf9-4ad7-90be-0f58348c80a3\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-e0907d-74b7bd8f9f-pdk27" Nov 03 01:59:09.915282 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:09.915249 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55jbp\" (UniqueName: \"kubernetes.io/projected/653ed959-fdf9-4ad7-90be-0f58348c80a3-kube-api-access-55jbp\") pod \"payload-processing-e2e-praxis-b-e0907d-74b7bd8f9f-pdk27\" (UID: \"653ed959-fdf9-4ad7-90be-0f58348c80a3\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-e0907d-74b7bd8f9f-pdk27" Nov 03 01:59:09.983945 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:09.983863 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-e0907d-74b7bd8f9f-pdk27" Nov 03 01:59:10.359387 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:59:10.359336 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod653ed959_fdf9_4ad7_90be_0f58348c80a3.slice/crio-2e932723d19d88adfbe0d17f3b0eaf8a40691500413d86e6d8a04fce2588a1dc WatchSource:0}: Error finding container 2e932723d19d88adfbe0d17f3b0eaf8a40691500413d86e6d8a04fce2588a1dc: Status 404 returned error can't find the container with id 2e932723d19d88adfbe0d17f3b0eaf8a40691500413d86e6d8a04fce2588a1dc Nov 03 01:59:10.362635 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:10.362614 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-e0907d-74b7bd8f9f-pdk27"] Nov 03 01:59:10.386268 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:10.386230 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-f95773-79bdcd49ff-mvz76" podUID="c31e0981-3a7c-4af3-9108-db5956c52d19" containerName="payload-processing" containerID="cri-o://729a5d1511d7b9871ea37cf11ab0a327066d54deddd04a5ab0977a35f84e5ba7" gracePeriod=25 Nov 03 01:59:10.638238 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:10.638206 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f95773-79bdcd49ff-mvz76" Nov 03 01:59:10.640619 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:10.640592 5437 generic.go:356] "Generic (PLEG): container finished" podID="c31e0981-3a7c-4af3-9108-db5956c52d19" containerID="729a5d1511d7b9871ea37cf11ab0a327066d54deddd04a5ab0977a35f84e5ba7" exitCode=0 Nov 03 01:59:10.640732 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:10.640634 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f95773-79bdcd49ff-mvz76" Nov 03 01:59:10.640732 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:10.640670 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f95773-79bdcd49ff-mvz76" event={"ID":"c31e0981-3a7c-4af3-9108-db5956c52d19","Type":"ContainerDied","Data":"729a5d1511d7b9871ea37cf11ab0a327066d54deddd04a5ab0977a35f84e5ba7"} Nov 03 01:59:10.640732 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:10.640719 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f95773-79bdcd49ff-mvz76" event={"ID":"c31e0981-3a7c-4af3-9108-db5956c52d19","Type":"ContainerDied","Data":"d4dc0dbdfdb5c8158d76eb26c79cc32aa0b0d46545c11d2b7334348e5293da37"} Nov 03 01:59:10.640845 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:10.640744 5437 scope.go:122] "RemoveContainer" containerID="729a5d1511d7b9871ea37cf11ab0a327066d54deddd04a5ab0977a35f84e5ba7" Nov 03 01:59:10.642398 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:10.642344 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-e0907d-74b7bd8f9f-pdk27" event={"ID":"653ed959-fdf9-4ad7-90be-0f58348c80a3","Type":"ContainerStarted","Data":"97e434e0da02e2b9bfc8972e7dc4ab769765378c860171b3d5f587b93eaafba3"} Nov 03 01:59:10.642520 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:10.642408 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-e0907d-74b7bd8f9f-pdk27" event={"ID":"653ed959-fdf9-4ad7-90be-0f58348c80a3","Type":"ContainerStarted","Data":"2e932723d19d88adfbe0d17f3b0eaf8a40691500413d86e6d8a04fce2588a1dc"} Nov 03 01:59:10.642520 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:10.642438 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-e0907d-74b7bd8f9f-pdk27" Nov 03 01:59:10.650677 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:10.650658 5437 scope.go:122] "RemoveContainer" containerID="729a5d1511d7b9871ea37cf11ab0a327066d54deddd04a5ab0977a35f84e5ba7" Nov 03 01:59:10.651004 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:59:10.650982 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"729a5d1511d7b9871ea37cf11ab0a327066d54deddd04a5ab0977a35f84e5ba7\": container with ID starting with 729a5d1511d7b9871ea37cf11ab0a327066d54deddd04a5ab0977a35f84e5ba7 not found: ID does not exist" containerID="729a5d1511d7b9871ea37cf11ab0a327066d54deddd04a5ab0977a35f84e5ba7" Nov 03 01:59:10.651109 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:10.651014 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"729a5d1511d7b9871ea37cf11ab0a327066d54deddd04a5ab0977a35f84e5ba7"} err="failed to get container status \"729a5d1511d7b9871ea37cf11ab0a327066d54deddd04a5ab0977a35f84e5ba7\": rpc error: code = NotFound desc = could not find container \"729a5d1511d7b9871ea37cf11ab0a327066d54deddd04a5ab0977a35f84e5ba7\": container with ID starting with 729a5d1511d7b9871ea37cf11ab0a327066d54deddd04a5ab0977a35f84e5ba7 not found: ID does not exist" Nov 03 01:59:10.762620 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:10.762591 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c31e0981-3a7c-4af3-9108-db5956c52d19-kube-api-access-lgw4h\" (UniqueName: \"kubernetes.io/projected/c31e0981-3a7c-4af3-9108-db5956c52d19-kube-api-access-lgw4h\") pod \"c31e0981-3a7c-4af3-9108-db5956c52d19\" (UID: \"c31e0981-3a7c-4af3-9108-db5956c52d19\") " Nov 03 01:59:10.762806 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:10.762656 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c31e0981-3a7c-4af3-9108-db5956c52d19-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c31e0981-3a7c-4af3-9108-db5956c52d19-plugins-config-volume\") pod \"c31e0981-3a7c-4af3-9108-db5956c52d19\" (UID: \"c31e0981-3a7c-4af3-9108-db5956c52d19\") " Nov 03 01:59:10.763105 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:10.763073 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c31e0981-3a7c-4af3-9108-db5956c52d19-plugins-config-volume" pod "c31e0981-3a7c-4af3-9108-db5956c52d19" (UID: "c31e0981-3a7c-4af3-9108-db5956c52d19"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:59:10.763397 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:10.763335 5437 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c31e0981-3a7c-4af3-9108-db5956c52d19-plugins-config-volume\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:59:10.765223 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:10.765199 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c31e0981-3a7c-4af3-9108-db5956c52d19-kube-api-access-lgw4h" pod "c31e0981-3a7c-4af3-9108-db5956c52d19" (UID: "c31e0981-3a7c-4af3-9108-db5956c52d19"). InnerVolumeSpecName "kube-api-access-lgw4h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:59:10.864090 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:10.864055 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgw4h\" (UniqueName: \"kubernetes.io/projected/c31e0981-3a7c-4af3-9108-db5956c52d19-kube-api-access-lgw4h\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:59:10.879708 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:10.879656 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-e0907d-74b7bd8f9f-pdk27" podStartSLOduration=1.8796426099999999 podStartE2EDuration="1.87964261s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:59:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:59:10.877472571 +0000 UTC m=+2432.906954867" watchObservedRunningTime="2026-11-03 01:59:10.87964261 +0000 UTC m=+2432.909124904" Nov 03 01:59:11.068869 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:11.068827 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f95773-79bdcd49ff-mvz76"] Nov 03 01:59:11.072836 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:11.072807 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f95773-79bdcd49ff-mvz76"] Nov 03 01:59:11.220689 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:11.220587 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:59:11.514020 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:11.513929 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:59:11.722739 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:11.722700 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:59:12.092667 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:12.092636 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c31e0981-3a7c-4af3-9108-db5956c52d19" path="/var/lib/kubelet/pods/c31e0981-3a7c-4af3-9108-db5956c52d19/volumes" Nov 03 01:59:14.034672 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:59:14.034630 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 01:59:14.574448 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:14.574414 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:59:15.295128 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:15.295103 5437 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-ghxht_c4e1a934-d917-417b-aea3-d432d2cff990/payload-processing/0.log" Nov 03 01:59:15.563167 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:15.563141 5437 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-ghxht_c4e1a934-d917-417b-aea3-d432d2cff990/payload-processing/0.log" Nov 03 01:59:16.145127 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:16.145091 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:59:16.383922 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:16.383882 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:59:16.650656 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:16.650621 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-db583d-7cf47dd48c-lrzwz"] Nov 03 01:59:16.651187 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:16.651174 5437 cpu_manager.go:414] "containerMap: removing container" podUID="c31e0981-3a7c-4af3-9108-db5956c52d19" containerName="payload-processing" Nov 03 01:59:16.651232 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:16.651188 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c31e0981-3a7c-4af3-9108-db5956c52d19" containerName="payload-processing" Nov 03 01:59:16.651340 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:16.651331 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="c31e0981-3a7c-4af3-9108-db5956c52d19" containerName="payload-processing" Nov 03 01:59:16.654806 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:16.654787 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-db583d-7cf47dd48c-lrzwz" Nov 03 01:59:16.708417 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:16.708387 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-db583d-7cf47dd48c-lrzwz"] Nov 03 01:59:16.735180 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:16.735142 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wz6l9\" (UniqueName: \"kubernetes.io/projected/768f8efe-c962-4d38-8743-a835df279bd7-kube-api-access-wz6l9\") pod \"payload-processing-e2e-ipp-b-db583d-7cf47dd48c-lrzwz\" (UID: \"768f8efe-c962-4d38-8743-a835df279bd7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-db583d-7cf47dd48c-lrzwz" Nov 03 01:59:16.735396 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:16.735340 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/768f8efe-c962-4d38-8743-a835df279bd7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-db583d-7cf47dd48c-lrzwz\" (UID: \"768f8efe-c962-4d38-8743-a835df279bd7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-db583d-7cf47dd48c-lrzwz" Nov 03 01:59:16.756759 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:16.756721 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-db583d\"" Nov 03 01:59:16.757039 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:16.757016 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-db583d-dockercfg-8jklj\"" Nov 03 01:59:16.836314 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:16.836239 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/768f8efe-c962-4d38-8743-a835df279bd7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-db583d-7cf47dd48c-lrzwz\" (UID: \"768f8efe-c962-4d38-8743-a835df279bd7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-db583d-7cf47dd48c-lrzwz" Nov 03 01:59:16.836498 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:16.836317 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wz6l9\" (UniqueName: \"kubernetes.io/projected/768f8efe-c962-4d38-8743-a835df279bd7-kube-api-access-wz6l9\") pod \"payload-processing-e2e-ipp-b-db583d-7cf47dd48c-lrzwz\" (UID: \"768f8efe-c962-4d38-8743-a835df279bd7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-db583d-7cf47dd48c-lrzwz" Nov 03 01:59:16.836979 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:16.836957 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/768f8efe-c962-4d38-8743-a835df279bd7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-db583d-7cf47dd48c-lrzwz\" (UID: \"768f8efe-c962-4d38-8743-a835df279bd7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-db583d-7cf47dd48c-lrzwz" Nov 03 01:59:16.899887 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:16.899852 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wz6l9\" (UniqueName: \"kubernetes.io/projected/768f8efe-c962-4d38-8743-a835df279bd7-kube-api-access-wz6l9\") pod \"payload-processing-e2e-ipp-b-db583d-7cf47dd48c-lrzwz\" (UID: \"768f8efe-c962-4d38-8743-a835df279bd7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-db583d-7cf47dd48c-lrzwz" Nov 03 01:59:16.968420 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:16.968308 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-db583d-7cf47dd48c-lrzwz" Nov 03 01:59:17.187101 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:17.187058 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:59:17.339526 ip-10-0-136-16 kubenswrapper[5437]: W1103 01:59:17.339490 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod768f8efe_c962_4d38_8743_a835df279bd7.slice/crio-eba1dac4798fd420fa66db7e58a3a6fb90c8f5546eff1c61125537fe86698401 WatchSource:0}: Error finding container eba1dac4798fd420fa66db7e58a3a6fb90c8f5546eff1c61125537fe86698401: Status 404 returned error can't find the container with id eba1dac4798fd420fa66db7e58a3a6fb90c8f5546eff1c61125537fe86698401 Nov 03 01:59:17.342256 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:17.342232 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-db583d-7cf47dd48c-lrzwz"] Nov 03 01:59:17.688069 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:17.687979 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-db583d-7cf47dd48c-lrzwz" event={"ID":"768f8efe-c962-4d38-8743-a835df279bd7","Type":"ContainerStarted","Data":"771b10b3356db42e6ec49621f7098d1294dc53e3e2cbe96222a3e0cd257f0dec"} Nov 03 01:59:17.688069 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:17.688031 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-db583d-7cf47dd48c-lrzwz" event={"ID":"768f8efe-c962-4d38-8743-a835df279bd7","Type":"ContainerStarted","Data":"eba1dac4798fd420fa66db7e58a3a6fb90c8f5546eff1c61125537fe86698401"} Nov 03 01:59:17.688549 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:17.688188 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-db583d-7cf47dd48c-lrzwz" Nov 03 01:59:17.808746 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:17.808692 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-db583d-7cf47dd48c-lrzwz" podStartSLOduration=1.8086795919999998 podStartE2EDuration="1.808679592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:59:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:59:17.807276942 +0000 UTC m=+2439.836759237" watchObservedRunningTime="2026-11-03 01:59:17.808679592 +0000 UTC m=+2439.838161898" Nov 03 01:59:19.430615 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:19.430574 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:59:21.652825 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:21.652795 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-e0907d-74b7bd8f9f-pdk27" Nov 03 01:59:22.089519 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:22.089489 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-b177c9/ipp-route-b177c9-kserve-7b945c8cfc-gk9vl"] Nov 03 01:59:22.221749 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:22.221710 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:59:24.168897 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:24.168846 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:59:28.697903 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:28.697871 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-db583d-7cf47dd48c-lrzwz" Nov 03 01:59:28.947031 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:28.946988 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:59:33.154961 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:33.154926 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:59:34.808670 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:34.808629 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:59:37.110243 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:37.110188 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-b177c9/ipp-route-b177c9-kserve-7b945c8cfc-gk9vl" podUID="b29aa32a-3985-4f47-906c-e27b88d21617" containerName="main" containerID="cri-o://d02316ce33689ab54f068cf1f43d47fc7d889b89ecff9fcb769bcfc2f22194f9" gracePeriod=45 Nov 03 01:59:37.124192 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:37.124136 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:59:37.381025 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:37.380990 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-b177c9/ipp-route-b177c9-kserve-7b945c8cfc-gk9vl" Nov 03 01:59:37.459089 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:37.459048 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:59:37.478854 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:37.478812 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b29aa32a-3985-4f47-906c-e27b88d21617-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b29aa32a-3985-4f47-906c-e27b88d21617-tmp-dir\") pod \"b29aa32a-3985-4f47-906c-e27b88d21617\" (UID: \"b29aa32a-3985-4f47-906c-e27b88d21617\") " Nov 03 01:59:37.479022 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:37.478873 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b29aa32a-3985-4f47-906c-e27b88d21617-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b29aa32a-3985-4f47-906c-e27b88d21617-model-cache\") pod \"b29aa32a-3985-4f47-906c-e27b88d21617\" (UID: \"b29aa32a-3985-4f47-906c-e27b88d21617\") " Nov 03 01:59:37.479022 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:37.478938 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b29aa32a-3985-4f47-906c-e27b88d21617-kube-api-access-t2jdz\" (UniqueName: \"kubernetes.io/projected/b29aa32a-3985-4f47-906c-e27b88d21617-kube-api-access-t2jdz\") pod \"b29aa32a-3985-4f47-906c-e27b88d21617\" (UID: \"b29aa32a-3985-4f47-906c-e27b88d21617\") " Nov 03 01:59:37.479110 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:37.479067 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b29aa32a-3985-4f47-906c-e27b88d21617-tls-certs\" (UniqueName: \"kubernetes.io/secret/b29aa32a-3985-4f47-906c-e27b88d21617-tls-certs\") pod \"b29aa32a-3985-4f47-906c-e27b88d21617\" (UID: \"b29aa32a-3985-4f47-906c-e27b88d21617\") " Nov 03 01:59:37.479110 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:37.479101 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b29aa32a-3985-4f47-906c-e27b88d21617-home\" (UniqueName: \"kubernetes.io/empty-dir/b29aa32a-3985-4f47-906c-e27b88d21617-home\") pod \"b29aa32a-3985-4f47-906c-e27b88d21617\" (UID: \"b29aa32a-3985-4f47-906c-e27b88d21617\") " Nov 03 01:59:37.479214 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:37.479160 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b29aa32a-3985-4f47-906c-e27b88d21617-tmp-dir" pod "b29aa32a-3985-4f47-906c-e27b88d21617" (UID: "b29aa32a-3985-4f47-906c-e27b88d21617"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:59:37.479214 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:37.479181 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b29aa32a-3985-4f47-906c-e27b88d21617-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b29aa32a-3985-4f47-906c-e27b88d21617-dshm\") pod \"b29aa32a-3985-4f47-906c-e27b88d21617\" (UID: \"b29aa32a-3985-4f47-906c-e27b88d21617\") " Nov 03 01:59:37.479214 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:37.479153 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b29aa32a-3985-4f47-906c-e27b88d21617-model-cache" pod "b29aa32a-3985-4f47-906c-e27b88d21617" (UID: "b29aa32a-3985-4f47-906c-e27b88d21617"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:59:37.479428 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:37.479391 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b29aa32a-3985-4f47-906c-e27b88d21617-home" pod "b29aa32a-3985-4f47-906c-e27b88d21617" (UID: "b29aa32a-3985-4f47-906c-e27b88d21617"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:59:37.479586 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:37.479565 5437 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b29aa32a-3985-4f47-906c-e27b88d21617-home\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:59:37.479586 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:37.479583 5437 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b29aa32a-3985-4f47-906c-e27b88d21617-tmp-dir\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:59:37.479586 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:37.479592 5437 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b29aa32a-3985-4f47-906c-e27b88d21617-model-cache\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:59:37.481381 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:37.481341 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b29aa32a-3985-4f47-906c-e27b88d21617-dshm" pod "b29aa32a-3985-4f47-906c-e27b88d21617" (UID: "b29aa32a-3985-4f47-906c-e27b88d21617"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:59:37.481381 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:37.481354 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b29aa32a-3985-4f47-906c-e27b88d21617-tls-certs" pod "b29aa32a-3985-4f47-906c-e27b88d21617" (UID: "b29aa32a-3985-4f47-906c-e27b88d21617"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:59:37.481532 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:37.481434 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b29aa32a-3985-4f47-906c-e27b88d21617-kube-api-access-t2jdz" pod "b29aa32a-3985-4f47-906c-e27b88d21617" (UID: "b29aa32a-3985-4f47-906c-e27b88d21617"). InnerVolumeSpecName "kube-api-access-t2jdz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:59:37.581195 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:37.581156 5437 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b29aa32a-3985-4f47-906c-e27b88d21617-dshm\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:59:37.581195 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:37.581191 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2jdz\" (UniqueName: \"kubernetes.io/projected/b29aa32a-3985-4f47-906c-e27b88d21617-kube-api-access-t2jdz\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:59:37.581469 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:37.581215 5437 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b29aa32a-3985-4f47-906c-e27b88d21617-tls-certs\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:59:37.815692 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:37.815658 5437 generic.go:356] "Generic (PLEG): container finished" podID="b29aa32a-3985-4f47-906c-e27b88d21617" containerID="d02316ce33689ab54f068cf1f43d47fc7d889b89ecff9fcb769bcfc2f22194f9" exitCode=0 Nov 03 01:59:37.815894 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:37.815705 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-b177c9/ipp-route-b177c9-kserve-7b945c8cfc-gk9vl" Nov 03 01:59:37.815894 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:37.815737 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-b177c9/ipp-route-b177c9-kserve-7b945c8cfc-gk9vl" event={"ID":"b29aa32a-3985-4f47-906c-e27b88d21617","Type":"ContainerDied","Data":"d02316ce33689ab54f068cf1f43d47fc7d889b89ecff9fcb769bcfc2f22194f9"} Nov 03 01:59:37.815894 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:37.815787 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-b177c9/ipp-route-b177c9-kserve-7b945c8cfc-gk9vl" event={"ID":"b29aa32a-3985-4f47-906c-e27b88d21617","Type":"ContainerDied","Data":"46b4652eb3fc84d3d146115a78f81f2c849e08465260baf5091a29aa0b25f4fb"} Nov 03 01:59:37.815894 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:37.815808 5437 scope.go:122] "RemoveContainer" containerID="d02316ce33689ab54f068cf1f43d47fc7d889b89ecff9fcb769bcfc2f22194f9" Nov 03 01:59:37.827824 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:37.827797 5437 scope.go:122] "RemoveContainer" containerID="d02316ce33689ab54f068cf1f43d47fc7d889b89ecff9fcb769bcfc2f22194f9" Nov 03 01:59:37.828194 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:59:37.828174 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d02316ce33689ab54f068cf1f43d47fc7d889b89ecff9fcb769bcfc2f22194f9\": container with ID starting with d02316ce33689ab54f068cf1f43d47fc7d889b89ecff9fcb769bcfc2f22194f9 not found: ID does not exist" containerID="d02316ce33689ab54f068cf1f43d47fc7d889b89ecff9fcb769bcfc2f22194f9" Nov 03 01:59:37.828271 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:37.828207 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d02316ce33689ab54f068cf1f43d47fc7d889b89ecff9fcb769bcfc2f22194f9"} err="failed to get container status \"d02316ce33689ab54f068cf1f43d47fc7d889b89ecff9fcb769bcfc2f22194f9\": rpc error: code = NotFound desc = could not find container \"d02316ce33689ab54f068cf1f43d47fc7d889b89ecff9fcb769bcfc2f22194f9\": container with ID starting with d02316ce33689ab54f068cf1f43d47fc7d889b89ecff9fcb769bcfc2f22194f9 not found: ID does not exist" Nov 03 01:59:37.987422 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:37.987387 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-b177c9/ipp-route-b177c9-kserve-7b945c8cfc-gk9vl"] Nov 03 01:59:37.992682 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:37.992652 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-b177c9/ipp-route-b177c9-kserve-7b945c8cfc-gk9vl"] Nov 03 01:59:38.090792 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:38.090707 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b29aa32a-3985-4f47-906c-e27b88d21617" path="/var/lib/kubelet/pods/b29aa32a-3985-4f47-906c-e27b88d21617/volumes" Nov 03 01:59:38.745941 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:38.745900 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:59:39.275743 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:39.275705 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:59:39.805222 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:39.805183 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:59:44.035219 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:59:44.035180 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 01:59:45.058683 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:45.058645 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:59:45.690167 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:45.690126 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:59:47.414593 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:47.414554 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:59:49.784919 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:49.784879 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:59:51.771248 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:51.771197 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:59:52.010450 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:52.010413 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:59:52.880852 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:52.880813 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-e2db44-65b559dccf-d5wgm"] Nov 03 01:59:53.173905 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:53.173792 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:59:53.419739 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:53.419705 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:59:54.428805 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:54.428767 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:59:56.774591 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:56.774553 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:59:57.640249 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:57.640209 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:59:57.881888 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:57.881845 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e2db44-65b559dccf-d5wgm" podUID="c1375bf2-794e-4412-be74-5bd983bd4ee6" containerName="payload-pre-processing" containerID="cri-o://cd24e1ac0cc70534bed6e836570a2416a0b199c370590f36ccec229600688f8e" gracePeriod=25 Nov 03 01:59:58.138717 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:58.138692 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e2db44-65b559dccf-d5wgm" Nov 03 01:59:58.229423 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:58.229330 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1375bf2-794e-4412-be74-5bd983bd4ee6-kube-api-access-b9g7s\" (UniqueName: \"kubernetes.io/projected/c1375bf2-794e-4412-be74-5bd983bd4ee6-kube-api-access-b9g7s\") pod \"c1375bf2-794e-4412-be74-5bd983bd4ee6\" (UID: \"c1375bf2-794e-4412-be74-5bd983bd4ee6\") " Nov 03 01:59:58.229602 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:58.229441 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1375bf2-794e-4412-be74-5bd983bd4ee6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1375bf2-794e-4412-be74-5bd983bd4ee6-plugins-config-volume\") pod \"c1375bf2-794e-4412-be74-5bd983bd4ee6\" (UID: \"c1375bf2-794e-4412-be74-5bd983bd4ee6\") " Nov 03 01:59:58.229859 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:58.229829 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1375bf2-794e-4412-be74-5bd983bd4ee6-plugins-config-volume" pod "c1375bf2-794e-4412-be74-5bd983bd4ee6" (UID: "c1375bf2-794e-4412-be74-5bd983bd4ee6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:59:58.231499 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:58.231470 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1375bf2-794e-4412-be74-5bd983bd4ee6-kube-api-access-b9g7s" pod "c1375bf2-794e-4412-be74-5bd983bd4ee6" (UID: "c1375bf2-794e-4412-be74-5bd983bd4ee6"). InnerVolumeSpecName "kube-api-access-b9g7s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:59:58.330657 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:58.330622 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9g7s\" (UniqueName: \"kubernetes.io/projected/c1375bf2-794e-4412-be74-5bd983bd4ee6-kube-api-access-b9g7s\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:59:58.330657 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:58.330655 5437 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1375bf2-794e-4412-be74-5bd983bd4ee6-plugins-config-volume\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:59:58.939217 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:58.939181 5437 generic.go:356] "Generic (PLEG): container finished" podID="c1375bf2-794e-4412-be74-5bd983bd4ee6" containerID="cd24e1ac0cc70534bed6e836570a2416a0b199c370590f36ccec229600688f8e" exitCode=0 Nov 03 01:59:58.939720 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:58.939235 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e2db44-65b559dccf-d5wgm" Nov 03 01:59:58.939720 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:58.939264 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e2db44-65b559dccf-d5wgm" event={"ID":"c1375bf2-794e-4412-be74-5bd983bd4ee6","Type":"ContainerDied","Data":"cd24e1ac0cc70534bed6e836570a2416a0b199c370590f36ccec229600688f8e"} Nov 03 01:59:58.939720 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:58.939319 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e2db44-65b559dccf-d5wgm" event={"ID":"c1375bf2-794e-4412-be74-5bd983bd4ee6","Type":"ContainerDied","Data":"26ba4c315f0d6e1222996c5026d73d2ccfe125a9c33c2cd22e928eb41f470dc5"} Nov 03 01:59:58.939720 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:58.939342 5437 scope.go:122] "RemoveContainer" containerID="cd24e1ac0cc70534bed6e836570a2416a0b199c370590f36ccec229600688f8e" Nov 03 01:59:58.949838 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:58.949817 5437 scope.go:122] "RemoveContainer" containerID="cd24e1ac0cc70534bed6e836570a2416a0b199c370590f36ccec229600688f8e" Nov 03 01:59:58.950140 ip-10-0-136-16 kubenswrapper[5437]: E1103 01:59:58.950119 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd24e1ac0cc70534bed6e836570a2416a0b199c370590f36ccec229600688f8e\": container with ID starting with cd24e1ac0cc70534bed6e836570a2416a0b199c370590f36ccec229600688f8e not found: ID does not exist" containerID="cd24e1ac0cc70534bed6e836570a2416a0b199c370590f36ccec229600688f8e" Nov 03 01:59:58.950209 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:58.950148 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd24e1ac0cc70534bed6e836570a2416a0b199c370590f36ccec229600688f8e"} err="failed to get container status \"cd24e1ac0cc70534bed6e836570a2416a0b199c370590f36ccec229600688f8e\": rpc error: code = NotFound desc = could not find container \"cd24e1ac0cc70534bed6e836570a2416a0b199c370590f36ccec229600688f8e\": container with ID starting with cd24e1ac0cc70534bed6e836570a2416a0b199c370590f36ccec229600688f8e not found: ID does not exist" Nov 03 01:59:59.015296 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:59.015247 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 01:59:59.110128 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:59.110092 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-e2db44-65b559dccf-d5wgm"] Nov 03 01:59:59.115813 ip-10-0-136-16 kubenswrapper[5437]: I1103 01:59:59.115785 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-e2db44-65b559dccf-d5wgm"] Nov 03 02:00:00.037617 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:00.037575 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:00:00.089151 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:00.089120 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1375bf2-794e-4412-be74-5bd983bd4ee6" path="/var/lib/kubelet/pods/c1375bf2-794e-4412-be74-5bd983bd4ee6/volumes" Nov 03 02:00:00.188713 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:00.188673 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e1793d-29894520-t47ng"] Nov 03 02:00:00.189416 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:00.189394 5437 cpu_manager.go:414] "containerMap: removing container" podUID="b29aa32a-3985-4f47-906c-e27b88d21617" containerName="main" Nov 03 02:00:00.189416 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:00.189415 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b29aa32a-3985-4f47-906c-e27b88d21617" containerName="main" Nov 03 02:00:00.189583 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:00.189442 5437 cpu_manager.go:414] "containerMap: removing container" podUID="c1375bf2-794e-4412-be74-5bd983bd4ee6" containerName="payload-pre-processing" Nov 03 02:00:00.189583 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:00.189447 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1375bf2-794e-4412-be74-5bd983bd4ee6" containerName="payload-pre-processing" Nov 03 02:00:00.189583 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:00.189550 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1375bf2-794e-4412-be74-5bd983bd4ee6" containerName="payload-pre-processing" Nov 03 02:00:00.189583 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:00.189566 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="b29aa32a-3985-4f47-906c-e27b88d21617" containerName="main" Nov 03 02:00:00.194230 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:00.194207 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894520-bxpgh"] Nov 03 02:00:00.194358 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:00.194335 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e1793d-29894520-t47ng" Nov 03 02:00:00.199215 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:00.199192 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894520-bxpgh" Nov 03 02:00:00.216077 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:00.216052 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41aa5f-29894520-pspxw"] Nov 03 02:00:00.220047 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:00.220028 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41aa5f-29894520-pspxw" Nov 03 02:00:00.251257 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:00.251223 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56f66e18-88ac-4524-aee2-ebec8cce6dd8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-41aa5f-29894520-pspxw\" (UID: \"56f66e18-88ac-4524-aee2-ebec8cce6dd8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41aa5f-29894520-pspxw" Nov 03 02:00:00.251593 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:00.251568 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pm9jq\" (UniqueName: \"kubernetes.io/projected/cb08db38-b50e-4488-ada1-ffce3cf76fd1-kube-api-access-pm9jq\") pod \"maas-api-key-cleanup-29894520-bxpgh\" (UID: \"cb08db38-b50e-4488-ada1-ffce3cf76fd1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894520-bxpgh" Nov 03 02:00:00.251771 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:00.251753 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb08db38-b50e-4488-ada1-ffce3cf76fd1-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894520-bxpgh\" (UID: \"cb08db38-b50e-4488-ada1-ffce3cf76fd1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894520-bxpgh" Nov 03 02:00:00.251927 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:00.251909 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86c42961-145d-4e9d-b66e-a0bc06cc6065-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-e1793d-29894520-t47ng\" (UID: \"86c42961-145d-4e9d-b66e-a0bc06cc6065\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e1793d-29894520-t47ng" Nov 03 02:00:00.252059 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:00.252039 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8qsr\" (UniqueName: \"kubernetes.io/projected/86c42961-145d-4e9d-b66e-a0bc06cc6065-kube-api-access-d8qsr\") pod \"maas-api-key-cleanup-e2e-worker-w1-e1793d-29894520-t47ng\" (UID: \"86c42961-145d-4e9d-b66e-a0bc06cc6065\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e1793d-29894520-t47ng" Nov 03 02:00:00.252176 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:00.252160 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rg47v\" (UniqueName: \"kubernetes.io/projected/56f66e18-88ac-4524-aee2-ebec8cce6dd8-kube-api-access-rg47v\") pod \"maas-api-key-cleanup-e2e-worker-w3-41aa5f-29894520-pspxw\" (UID: \"56f66e18-88ac-4524-aee2-ebec8cce6dd8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41aa5f-29894520-pspxw" Nov 03 02:00:00.314260 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:00.314224 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e1793d-29894520-t47ng"] Nov 03 02:00:00.319492 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:00.319447 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894520-bxpgh"] Nov 03 02:00:00.322877 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:00.322846 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41aa5f-29894520-pspxw"] Nov 03 02:00:00.353197 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:00.353160 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56f66e18-88ac-4524-aee2-ebec8cce6dd8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-41aa5f-29894520-pspxw\" (UID: \"56f66e18-88ac-4524-aee2-ebec8cce6dd8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41aa5f-29894520-pspxw" Nov 03 02:00:00.353413 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:00.353239 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pm9jq\" (UniqueName: \"kubernetes.io/projected/cb08db38-b50e-4488-ada1-ffce3cf76fd1-kube-api-access-pm9jq\") pod \"maas-api-key-cleanup-29894520-bxpgh\" (UID: \"cb08db38-b50e-4488-ada1-ffce3cf76fd1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894520-bxpgh" Nov 03 02:00:00.353413 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:00.353333 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb08db38-b50e-4488-ada1-ffce3cf76fd1-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894520-bxpgh\" (UID: \"cb08db38-b50e-4488-ada1-ffce3cf76fd1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894520-bxpgh" Nov 03 02:00:00.353553 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:00.353417 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86c42961-145d-4e9d-b66e-a0bc06cc6065-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-e1793d-29894520-t47ng\" (UID: \"86c42961-145d-4e9d-b66e-a0bc06cc6065\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e1793d-29894520-t47ng" Nov 03 02:00:00.353553 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:00.353463 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8qsr\" (UniqueName: \"kubernetes.io/projected/86c42961-145d-4e9d-b66e-a0bc06cc6065-kube-api-access-d8qsr\") pod \"maas-api-key-cleanup-e2e-worker-w1-e1793d-29894520-t47ng\" (UID: \"86c42961-145d-4e9d-b66e-a0bc06cc6065\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e1793d-29894520-t47ng" Nov 03 02:00:00.353674 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:00.353567 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rg47v\" (UniqueName: \"kubernetes.io/projected/56f66e18-88ac-4524-aee2-ebec8cce6dd8-kube-api-access-rg47v\") pod \"maas-api-key-cleanup-e2e-worker-w3-41aa5f-29894520-pspxw\" (UID: \"56f66e18-88ac-4524-aee2-ebec8cce6dd8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41aa5f-29894520-pspxw" Nov 03 02:00:00.353988 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:00.353959 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56f66e18-88ac-4524-aee2-ebec8cce6dd8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-41aa5f-29894520-pspxw\" (UID: \"56f66e18-88ac-4524-aee2-ebec8cce6dd8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41aa5f-29894520-pspxw" Nov 03 02:00:00.354263 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:00.354241 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86c42961-145d-4e9d-b66e-a0bc06cc6065-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-e1793d-29894520-t47ng\" (UID: \"86c42961-145d-4e9d-b66e-a0bc06cc6065\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e1793d-29894520-t47ng" Nov 03 02:00:00.354554 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:00.354533 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb08db38-b50e-4488-ada1-ffce3cf76fd1-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894520-bxpgh\" (UID: \"cb08db38-b50e-4488-ada1-ffce3cf76fd1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894520-bxpgh" Nov 03 02:00:00.362192 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:00.362164 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fgwsl\"" Nov 03 02:00:00.416233 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:00.416202 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8qsr\" (UniqueName: \"kubernetes.io/projected/86c42961-145d-4e9d-b66e-a0bc06cc6065-kube-api-access-d8qsr\") pod \"maas-api-key-cleanup-e2e-worker-w1-e1793d-29894520-t47ng\" (UID: \"86c42961-145d-4e9d-b66e-a0bc06cc6065\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e1793d-29894520-t47ng" Nov 03 02:00:00.416442 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:00.416384 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rg47v\" (UniqueName: \"kubernetes.io/projected/56f66e18-88ac-4524-aee2-ebec8cce6dd8-kube-api-access-rg47v\") pod \"maas-api-key-cleanup-e2e-worker-w3-41aa5f-29894520-pspxw\" (UID: \"56f66e18-88ac-4524-aee2-ebec8cce6dd8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41aa5f-29894520-pspxw" Nov 03 02:00:00.416691 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:00.416674 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pm9jq\" (UniqueName: \"kubernetes.io/projected/cb08db38-b50e-4488-ada1-ffce3cf76fd1-kube-api-access-pm9jq\") pod \"maas-api-key-cleanup-29894520-bxpgh\" (UID: \"cb08db38-b50e-4488-ada1-ffce3cf76fd1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894520-bxpgh" Nov 03 02:00:00.510827 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:00.510787 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e1793d-29894520-t47ng" Nov 03 02:00:00.517116 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:00.517096 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894520-bxpgh" Nov 03 02:00:00.533276 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:00.533219 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41aa5f-29894520-pspxw" Nov 03 02:00:00.890472 ip-10-0-136-16 kubenswrapper[5437]: W1103 02:00:00.890428 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86c42961_145d_4e9d_b66e_a0bc06cc6065.slice/crio-890efda6519a9dce1389b576545270c4ec9371b39d3987b980f2726b53060885 WatchSource:0}: Error finding container 890efda6519a9dce1389b576545270c4ec9371b39d3987b980f2726b53060885: Status 404 returned error can't find the container with id 890efda6519a9dce1389b576545270c4ec9371b39d3987b980f2726b53060885 Nov 03 02:00:00.893585 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:00.893563 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e1793d-29894520-t47ng"] Nov 03 02:00:00.938186 ip-10-0-136-16 kubenswrapper[5437]: W1103 02:00:00.938155 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb08db38_b50e_4488_ada1_ffce3cf76fd1.slice/crio-c31f68f5003cb115d1f122ea722c0e6ad9cbb5b6bd47960dea7e31948f6bd106 WatchSource:0}: Error finding container c31f68f5003cb115d1f122ea722c0e6ad9cbb5b6bd47960dea7e31948f6bd106: Status 404 returned error can't find the container with id c31f68f5003cb115d1f122ea722c0e6ad9cbb5b6bd47960dea7e31948f6bd106 Nov 03 02:00:00.939137 ip-10-0-136-16 kubenswrapper[5437]: W1103 02:00:00.939109 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56f66e18_88ac_4524_aee2_ebec8cce6dd8.slice/crio-f38f5852b14ef51fcc16722008bfbdf5ddc5fb82c441340b4e65b1ccf94d0279 WatchSource:0}: Error finding container f38f5852b14ef51fcc16722008bfbdf5ddc5fb82c441340b4e65b1ccf94d0279: Status 404 returned error can't find the container with id f38f5852b14ef51fcc16722008bfbdf5ddc5fb82c441340b4e65b1ccf94d0279 Nov 03 02:00:00.941711 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:00.941546 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894520-bxpgh"] Nov 03 02:00:00.944695 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:00.944672 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41aa5f-29894520-pspxw"] Nov 03 02:00:00.953695 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:00.953665 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41aa5f-29894520-pspxw" event={"ID":"56f66e18-88ac-4524-aee2-ebec8cce6dd8","Type":"ContainerStarted","Data":"f38f5852b14ef51fcc16722008bfbdf5ddc5fb82c441340b4e65b1ccf94d0279"} Nov 03 02:00:00.955311 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:00.955278 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894520-bxpgh" event={"ID":"cb08db38-b50e-4488-ada1-ffce3cf76fd1","Type":"ContainerStarted","Data":"c31f68f5003cb115d1f122ea722c0e6ad9cbb5b6bd47960dea7e31948f6bd106"} Nov 03 02:00:00.957013 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:00.956984 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e1793d-29894520-t47ng" event={"ID":"86c42961-145d-4e9d-b66e-a0bc06cc6065","Type":"ContainerStarted","Data":"890efda6519a9dce1389b576545270c4ec9371b39d3987b980f2726b53060885"} Nov 03 02:00:01.818642 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:01.818587 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:00:01.966193 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:01.966152 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e1793d-29894520-t47ng" event={"ID":"86c42961-145d-4e9d-b66e-a0bc06cc6065","Type":"ContainerStarted","Data":"0c2b140229c6d33515d3a96d691e396a9a8937509da0589c04041e48b56ecc95"} Nov 03 02:00:01.967802 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:01.967776 5437 generic.go:356] "Generic (PLEG): container finished" podID="56f66e18-88ac-4524-aee2-ebec8cce6dd8" containerID="43ee36342160554052f8a9436a5529717c1883036e9e9f2c0fe01c567dc83bf8" exitCode=60 Nov 03 02:00:01.967919 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:01.967853 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41aa5f-29894520-pspxw" event={"ID":"56f66e18-88ac-4524-aee2-ebec8cce6dd8","Type":"ContainerDied","Data":"43ee36342160554052f8a9436a5529717c1883036e9e9f2c0fe01c567dc83bf8"} Nov 03 02:00:01.968088 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:01.968073 5437 scope.go:122] "RemoveContainer" containerID="43ee36342160554052f8a9436a5529717c1883036e9e9f2c0fe01c567dc83bf8" Nov 03 02:00:01.969445 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:01.969425 5437 generic.go:356] "Generic (PLEG): container finished" podID="cb08db38-b50e-4488-ada1-ffce3cf76fd1" containerID="20b57c6c1846bc83caffab132d680cb96aa7796552d36114f78036c33c0ab810" exitCode=60 Nov 03 02:00:01.969559 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:01.969459 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894520-bxpgh" event={"ID":"cb08db38-b50e-4488-ada1-ffce3cf76fd1","Type":"ContainerDied","Data":"20b57c6c1846bc83caffab132d680cb96aa7796552d36114f78036c33c0ab810"} Nov 03 02:00:01.969782 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:01.969764 5437 scope.go:122] "RemoveContainer" containerID="20b57c6c1846bc83caffab132d680cb96aa7796552d36114f78036c33c0ab810" Nov 03 02:00:02.087103 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:02.086982 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e1793d-29894520-t47ng" podStartSLOduration=2.086967516 podStartE2EDuration="2.086967516s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:00:02.083925591 +0000 UTC m=+2484.113407886" watchObservedRunningTime="2026-11-03 02:00:02.086967516 +0000 UTC m=+2484.116449867" Nov 03 02:00:02.977818 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:02.977782 5437 generic.go:356] "Generic (PLEG): container finished" podID="56f66e18-88ac-4524-aee2-ebec8cce6dd8" containerID="dc967e1e332fb9768320cdc447f4b9113e1778e0df4d9669e5cf79659e18ff6d" exitCode=60 Nov 03 02:00:02.978304 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:02.977844 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41aa5f-29894520-pspxw" event={"ID":"56f66e18-88ac-4524-aee2-ebec8cce6dd8","Type":"ContainerDied","Data":"dc967e1e332fb9768320cdc447f4b9113e1778e0df4d9669e5cf79659e18ff6d"} Nov 03 02:00:02.978304 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:02.977952 5437 scope.go:122] "RemoveContainer" containerID="43ee36342160554052f8a9436a5529717c1883036e9e9f2c0fe01c567dc83bf8" Nov 03 02:00:02.978304 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:02.978289 5437 scope.go:122] "RemoveContainer" containerID="dc967e1e332fb9768320cdc447f4b9113e1778e0df4d9669e5cf79659e18ff6d" Nov 03 02:00:02.978656 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:00:02.978635 5437 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-41aa5f-29894520-pspxw_odh-ai-gateway-infra(56f66e18-88ac-4524-aee2-ebec8cce6dd8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41aa5f-29894520-pspxw" podUID="56f66e18-88ac-4524-aee2-ebec8cce6dd8" Nov 03 02:00:02.980065 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:02.980036 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894520-bxpgh" event={"ID":"cb08db38-b50e-4488-ada1-ffce3cf76fd1","Type":"ContainerStarted","Data":"c06adf88341d3e6cf6591218226001c530ee35f37f61970e8fe3ce6d3a0d3b8e"} Nov 03 02:00:02.981703 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:02.981682 5437 generic.go:356] "Generic (PLEG): container finished" podID="86c42961-145d-4e9d-b66e-a0bc06cc6065" containerID="0c2b140229c6d33515d3a96d691e396a9a8937509da0589c04041e48b56ecc95" exitCode=60 Nov 03 02:00:02.981790 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:02.981732 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e1793d-29894520-t47ng" event={"ID":"86c42961-145d-4e9d-b66e-a0bc06cc6065","Type":"ContainerDied","Data":"0c2b140229c6d33515d3a96d691e396a9a8937509da0589c04041e48b56ecc95"} Nov 03 02:00:02.982003 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:02.981987 5437 scope.go:122] "RemoveContainer" containerID="0c2b140229c6d33515d3a96d691e396a9a8937509da0589c04041e48b56ecc95" Nov 03 02:00:03.216448 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:03.216299 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894520-bxpgh" podStartSLOduration=3.216283098 podStartE2EDuration="3.216283098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:00:03.21482669 +0000 UTC m=+2485.244308986" watchObservedRunningTime="2026-11-03 02:00:03.216283098 +0000 UTC m=+2485.245765392" Nov 03 02:00:03.990471 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:03.990436 5437 generic.go:356] "Generic (PLEG): container finished" podID="86c42961-145d-4e9d-b66e-a0bc06cc6065" containerID="706853eae2b612c502eaef7cc71ad43f444c864a2c73f22b20db8c24df00894c" exitCode=60 Nov 03 02:00:03.990913 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:03.990521 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e1793d-29894520-t47ng" event={"ID":"86c42961-145d-4e9d-b66e-a0bc06cc6065","Type":"ContainerDied","Data":"706853eae2b612c502eaef7cc71ad43f444c864a2c73f22b20db8c24df00894c"} Nov 03 02:00:03.990913 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:03.990592 5437 scope.go:122] "RemoveContainer" containerID="0c2b140229c6d33515d3a96d691e396a9a8937509da0589c04041e48b56ecc95" Nov 03 02:00:03.990913 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:03.990824 5437 scope.go:122] "RemoveContainer" containerID="706853eae2b612c502eaef7cc71ad43f444c864a2c73f22b20db8c24df00894c" Nov 03 02:00:03.991182 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:00:03.991157 5437 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-e1793d-29894520-t47ng_odh-ai-gateway-infra(86c42961-145d-4e9d-b66e-a0bc06cc6065)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e1793d-29894520-t47ng" podUID="86c42961-145d-4e9d-b66e-a0bc06cc6065" Nov 03 02:00:03.992611 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:03.992584 5437 scope.go:122] "RemoveContainer" containerID="dc967e1e332fb9768320cdc447f4b9113e1778e0df4d9669e5cf79659e18ff6d" Nov 03 02:00:03.992855 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:00:03.992830 5437 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-41aa5f-29894520-pspxw_odh-ai-gateway-infra(56f66e18-88ac-4524-aee2-ebec8cce6dd8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41aa5f-29894520-pspxw" podUID="56f66e18-88ac-4524-aee2-ebec8cce6dd8" Nov 03 02:00:03.998483 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:03.998456 5437 generic.go:356] "Generic (PLEG): container finished" podID="cb08db38-b50e-4488-ada1-ffce3cf76fd1" containerID="c06adf88341d3e6cf6591218226001c530ee35f37f61970e8fe3ce6d3a0d3b8e" exitCode=60 Nov 03 02:00:03.998615 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:03.998505 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894520-bxpgh" event={"ID":"cb08db38-b50e-4488-ada1-ffce3cf76fd1","Type":"ContainerDied","Data":"c06adf88341d3e6cf6591218226001c530ee35f37f61970e8fe3ce6d3a0d3b8e"} Nov 03 02:00:03.998776 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:03.998761 5437 scope.go:122] "RemoveContainer" containerID="c06adf88341d3e6cf6591218226001c530ee35f37f61970e8fe3ce6d3a0d3b8e" Nov 03 02:00:03.998990 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:00:03.998974 5437 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894520-bxpgh_odh-ai-gateway-infra(cb08db38-b50e-4488-ada1-ffce3cf76fd1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894520-bxpgh" podUID="cb08db38-b50e-4488-ada1-ffce3cf76fd1" Nov 03 02:00:04.006869 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:04.006851 5437 scope.go:122] "RemoveContainer" containerID="20b57c6c1846bc83caffab132d680cb96aa7796552d36114f78036c33c0ab810" Nov 03 02:00:05.007079 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:05.007042 5437 scope.go:122] "RemoveContainer" containerID="c06adf88341d3e6cf6591218226001c530ee35f37f61970e8fe3ce6d3a0d3b8e" Nov 03 02:00:05.007589 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:00:05.007359 5437 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894520-bxpgh_odh-ai-gateway-infra(cb08db38-b50e-4488-ada1-ffce3cf76fd1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894520-bxpgh" podUID="cb08db38-b50e-4488-ada1-ffce3cf76fd1" Nov 03 02:00:05.008763 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:05.008745 5437 scope.go:122] "RemoveContainer" containerID="706853eae2b612c502eaef7cc71ad43f444c864a2c73f22b20db8c24df00894c" Nov 03 02:00:05.009025 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:00:05.009006 5437 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-e1793d-29894520-t47ng_odh-ai-gateway-infra(86c42961-145d-4e9d-b66e-a0bc06cc6065)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e1793d-29894520-t47ng" podUID="86c42961-145d-4e9d-b66e-a0bc06cc6065" Nov 03 02:00:06.041082 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:06.041042 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:00:06.660057 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:06.660018 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:00:07.184181 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:07.184130 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:00:08.961761 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:08.961721 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:00:10.893024 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:10.892984 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:00:11.407896 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:11.407853 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:00:12.162578 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:12.162523 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:00:14.034427 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:00:14.034392 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 02:00:14.082126 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:14.082095 5437 scope.go:122] "RemoveContainer" containerID="c06adf88341d3e6cf6591218226001c530ee35f37f61970e8fe3ce6d3a0d3b8e" Nov 03 02:00:14.243201 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:14.243160 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:00:14.719872 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:14.719788 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:00:14.922850 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:14.922806 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:00:15.078941 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:15.078906 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894520-bxpgh" event={"ID":"cb08db38-b50e-4488-ada1-ffce3cf76fd1","Type":"ContainerStarted","Data":"b0271b9b452230cb89179a9d3d1ba9a546a89b498a6d95f0ab644c1c9483c00f"} Nov 03 02:00:15.081768 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:15.081751 5437 scope.go:122] "RemoveContainer" containerID="706853eae2b612c502eaef7cc71ad43f444c864a2c73f22b20db8c24df00894c" Nov 03 02:00:15.125195 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:15.125166 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:00:16.081755 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:16.081727 5437 scope.go:122] "RemoveContainer" containerID="dc967e1e332fb9768320cdc447f4b9113e1778e0df4d9669e5cf79659e18ff6d" Nov 03 02:00:16.088575 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:16.088546 5437 generic.go:356] "Generic (PLEG): container finished" podID="86c42961-145d-4e9d-b66e-a0bc06cc6065" containerID="b1ee8518accfe82dbb7ea6f9641c7191dbc15f9b8ebbffebd54a6138d95862c0" exitCode=60 Nov 03 02:00:16.089770 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:16.089747 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e1793d-29894520-t47ng" event={"ID":"86c42961-145d-4e9d-b66e-a0bc06cc6065","Type":"ContainerDied","Data":"b1ee8518accfe82dbb7ea6f9641c7191dbc15f9b8ebbffebd54a6138d95862c0"} Nov 03 02:00:16.089891 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:16.089809 5437 scope.go:122] "RemoveContainer" containerID="706853eae2b612c502eaef7cc71ad43f444c864a2c73f22b20db8c24df00894c" Nov 03 02:00:16.090687 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:16.090158 5437 scope.go:122] "RemoveContainer" containerID="b1ee8518accfe82dbb7ea6f9641c7191dbc15f9b8ebbffebd54a6138d95862c0" Nov 03 02:00:16.090687 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:00:16.090456 5437 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-e1793d-29894520-t47ng_odh-ai-gateway-infra(86c42961-145d-4e9d-b66e-a0bc06cc6065)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e1793d-29894520-t47ng" podUID="86c42961-145d-4e9d-b66e-a0bc06cc6065" Nov 03 02:00:16.094488 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:16.091621 5437 generic.go:356] "Generic (PLEG): container finished" podID="cb08db38-b50e-4488-ada1-ffce3cf76fd1" containerID="b0271b9b452230cb89179a9d3d1ba9a546a89b498a6d95f0ab644c1c9483c00f" exitCode=60 Nov 03 02:00:16.094488 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:16.091717 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894520-bxpgh" event={"ID":"cb08db38-b50e-4488-ada1-ffce3cf76fd1","Type":"ContainerDied","Data":"b0271b9b452230cb89179a9d3d1ba9a546a89b498a6d95f0ab644c1c9483c00f"} Nov 03 02:00:16.094488 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:16.092062 5437 scope.go:122] "RemoveContainer" containerID="b0271b9b452230cb89179a9d3d1ba9a546a89b498a6d95f0ab644c1c9483c00f" Nov 03 02:00:16.094488 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:00:16.092301 5437 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29894520-bxpgh_odh-ai-gateway-infra(cb08db38-b50e-4488-ada1-ffce3cf76fd1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894520-bxpgh" podUID="cb08db38-b50e-4488-ada1-ffce3cf76fd1" Nov 03 02:00:16.115465 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:16.115432 5437 scope.go:122] "RemoveContainer" containerID="c06adf88341d3e6cf6591218226001c530ee35f37f61970e8fe3ce6d3a0d3b8e" Nov 03 02:00:16.208183 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:16.208145 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894520-bxpgh"] Nov 03 02:00:17.101889 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:17.101851 5437 generic.go:356] "Generic (PLEG): container finished" podID="56f66e18-88ac-4524-aee2-ebec8cce6dd8" containerID="fffc15b8ffc2bb2b75ffb0c415e5b7ea9002f3c6a88ff58931ca178b6b6c678d" exitCode=60 Nov 03 02:00:17.102334 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:17.101921 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41aa5f-29894520-pspxw" event={"ID":"56f66e18-88ac-4524-aee2-ebec8cce6dd8","Type":"ContainerDied","Data":"fffc15b8ffc2bb2b75ffb0c415e5b7ea9002f3c6a88ff58931ca178b6b6c678d"} Nov 03 02:00:17.102334 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:17.101974 5437 scope.go:122] "RemoveContainer" containerID="dc967e1e332fb9768320cdc447f4b9113e1778e0df4d9669e5cf79659e18ff6d" Nov 03 02:00:17.102473 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:17.102404 5437 scope.go:122] "RemoveContainer" containerID="fffc15b8ffc2bb2b75ffb0c415e5b7ea9002f3c6a88ff58931ca178b6b6c678d" Nov 03 02:00:17.102710 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:00:17.102688 5437 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-41aa5f-29894520-pspxw_odh-ai-gateway-infra(56f66e18-88ac-4524-aee2-ebec8cce6dd8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41aa5f-29894520-pspxw" podUID="56f66e18-88ac-4524-aee2-ebec8cce6dd8" Nov 03 02:00:17.220269 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:17.220189 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e1793d-29894520-t47ng"] Nov 03 02:00:17.279942 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:17.279915 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894520-bxpgh" Nov 03 02:00:17.375439 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:17.372825 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb08db38-b50e-4488-ada1-ffce3cf76fd1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb08db38-b50e-4488-ada1-ffce3cf76fd1-maas-api-service-ca\") pod \"cb08db38-b50e-4488-ada1-ffce3cf76fd1\" (UID: \"cb08db38-b50e-4488-ada1-ffce3cf76fd1\") " Nov 03 02:00:17.375439 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:17.372911 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb08db38-b50e-4488-ada1-ffce3cf76fd1-kube-api-access-pm9jq\" (UniqueName: \"kubernetes.io/projected/cb08db38-b50e-4488-ada1-ffce3cf76fd1-kube-api-access-pm9jq\") pod \"cb08db38-b50e-4488-ada1-ffce3cf76fd1\" (UID: \"cb08db38-b50e-4488-ada1-ffce3cf76fd1\") " Nov 03 02:00:17.375439 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:17.373602 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb08db38-b50e-4488-ada1-ffce3cf76fd1-maas-api-service-ca" pod "cb08db38-b50e-4488-ada1-ffce3cf76fd1" (UID: "cb08db38-b50e-4488-ada1-ffce3cf76fd1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:00:17.376829 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:17.376794 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb08db38-b50e-4488-ada1-ffce3cf76fd1-kube-api-access-pm9jq" pod "cb08db38-b50e-4488-ada1-ffce3cf76fd1" (UID: "cb08db38-b50e-4488-ada1-ffce3cf76fd1"). InnerVolumeSpecName "kube-api-access-pm9jq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:00:17.413712 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:17.413684 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e1793d-29894520-t47ng" Nov 03 02:00:17.475301 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:17.475220 5437 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb08db38-b50e-4488-ada1-ffce3cf76fd1-maas-api-service-ca\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:00:17.475301 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:17.475248 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pm9jq\" (UniqueName: \"kubernetes.io/projected/cb08db38-b50e-4488-ada1-ffce3cf76fd1-kube-api-access-pm9jq\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:00:17.576640 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:17.576606 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86c42961-145d-4e9d-b66e-a0bc06cc6065-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86c42961-145d-4e9d-b66e-a0bc06cc6065-maas-api-service-ca\") pod \"86c42961-145d-4e9d-b66e-a0bc06cc6065\" (UID: \"86c42961-145d-4e9d-b66e-a0bc06cc6065\") " Nov 03 02:00:17.576828 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:17.576816 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86c42961-145d-4e9d-b66e-a0bc06cc6065-kube-api-access-d8qsr\" (UniqueName: \"kubernetes.io/projected/86c42961-145d-4e9d-b66e-a0bc06cc6065-kube-api-access-d8qsr\") pod \"86c42961-145d-4e9d-b66e-a0bc06cc6065\" (UID: \"86c42961-145d-4e9d-b66e-a0bc06cc6065\") " Nov 03 02:00:17.576965 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:17.576941 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86c42961-145d-4e9d-b66e-a0bc06cc6065-maas-api-service-ca" pod "86c42961-145d-4e9d-b66e-a0bc06cc6065" (UID: "86c42961-145d-4e9d-b66e-a0bc06cc6065"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:00:17.577152 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:17.577139 5437 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86c42961-145d-4e9d-b66e-a0bc06cc6065-maas-api-service-ca\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:00:17.578973 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:17.578949 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86c42961-145d-4e9d-b66e-a0bc06cc6065-kube-api-access-d8qsr" pod "86c42961-145d-4e9d-b66e-a0bc06cc6065" (UID: "86c42961-145d-4e9d-b66e-a0bc06cc6065"). InnerVolumeSpecName "kube-api-access-d8qsr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:00:17.678229 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:17.678193 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8qsr\" (UniqueName: \"kubernetes.io/projected/86c42961-145d-4e9d-b66e-a0bc06cc6065-kube-api-access-d8qsr\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:00:18.122924 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:18.122887 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e1793d-29894520-t47ng" Nov 03 02:00:18.122924 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:18.122908 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e1793d-29894520-t47ng" event={"ID":"86c42961-145d-4e9d-b66e-a0bc06cc6065","Type":"ContainerDied","Data":"890efda6519a9dce1389b576545270c4ec9371b39d3987b980f2726b53060885"} Nov 03 02:00:18.123616 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:18.122979 5437 scope.go:122] "RemoveContainer" containerID="b1ee8518accfe82dbb7ea6f9641c7191dbc15f9b8ebbffebd54a6138d95862c0" Nov 03 02:00:18.129851 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:18.129474 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894520-bxpgh" event={"ID":"cb08db38-b50e-4488-ada1-ffce3cf76fd1","Type":"ContainerDied","Data":"c31f68f5003cb115d1f122ea722c0e6ad9cbb5b6bd47960dea7e31948f6bd106"} Nov 03 02:00:18.129851 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:18.129527 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894520-bxpgh" Nov 03 02:00:18.135604 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:18.135578 5437 scope.go:122] "RemoveContainer" containerID="b0271b9b452230cb89179a9d3d1ba9a546a89b498a6d95f0ab644c1c9483c00f" Nov 03 02:00:18.231903 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:18.231806 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41aa5f-29894520-pspxw"] Nov 03 02:00:18.297736 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:18.297700 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e1793d-29894520-t47ng"] Nov 03 02:00:18.306010 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:18.305979 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e1793d-29894520-t47ng"] Nov 03 02:00:18.400493 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:18.400465 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41aa5f-29894520-pspxw" Nov 03 02:00:18.418726 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:18.418690 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894520-bxpgh"] Nov 03 02:00:18.422286 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:18.422260 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894520-bxpgh"] Nov 03 02:00:18.589072 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:18.589036 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56f66e18-88ac-4524-aee2-ebec8cce6dd8-kube-api-access-rg47v\" (UniqueName: \"kubernetes.io/projected/56f66e18-88ac-4524-aee2-ebec8cce6dd8-kube-api-access-rg47v\") pod \"56f66e18-88ac-4524-aee2-ebec8cce6dd8\" (UID: \"56f66e18-88ac-4524-aee2-ebec8cce6dd8\") " Nov 03 02:00:18.589225 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:18.589083 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56f66e18-88ac-4524-aee2-ebec8cce6dd8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56f66e18-88ac-4524-aee2-ebec8cce6dd8-maas-api-service-ca\") pod \"56f66e18-88ac-4524-aee2-ebec8cce6dd8\" (UID: \"56f66e18-88ac-4524-aee2-ebec8cce6dd8\") " Nov 03 02:00:18.589489 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:18.589467 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56f66e18-88ac-4524-aee2-ebec8cce6dd8-maas-api-service-ca" pod "56f66e18-88ac-4524-aee2-ebec8cce6dd8" (UID: "56f66e18-88ac-4524-aee2-ebec8cce6dd8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:00:18.591187 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:18.591166 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56f66e18-88ac-4524-aee2-ebec8cce6dd8-kube-api-access-rg47v" pod "56f66e18-88ac-4524-aee2-ebec8cce6dd8" (UID: "56f66e18-88ac-4524-aee2-ebec8cce6dd8"). InnerVolumeSpecName "kube-api-access-rg47v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:00:18.690295 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:18.690258 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rg47v\" (UniqueName: \"kubernetes.io/projected/56f66e18-88ac-4524-aee2-ebec8cce6dd8-kube-api-access-rg47v\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:00:18.690295 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:18.690289 5437 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56f66e18-88ac-4524-aee2-ebec8cce6dd8-maas-api-service-ca\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:00:19.139963 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:19.139933 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41aa5f-29894520-pspxw" Nov 03 02:00:19.140484 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:19.139959 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41aa5f-29894520-pspxw" event={"ID":"56f66e18-88ac-4524-aee2-ebec8cce6dd8","Type":"ContainerDied","Data":"f38f5852b14ef51fcc16722008bfbdf5ddc5fb82c441340b4e65b1ccf94d0279"} Nov 03 02:00:19.140484 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:19.140023 5437 scope.go:122] "RemoveContainer" containerID="fffc15b8ffc2bb2b75ffb0c415e5b7ea9002f3c6a88ff58931ca178b6b6c678d" Nov 03 02:00:19.314684 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:19.314650 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41aa5f-29894520-pspxw"] Nov 03 02:00:19.317503 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:19.317473 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41aa5f-29894520-pspxw"] Nov 03 02:00:19.319612 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:19.319579 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:00:19.524841 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:19.524760 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:00:19.732561 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:19.732518 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:00:20.091194 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:20.091157 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56f66e18-88ac-4524-aee2-ebec8cce6dd8" path="/var/lib/kubelet/pods/56f66e18-88ac-4524-aee2-ebec8cce6dd8/volumes" Nov 03 02:00:20.091766 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:20.091743 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86c42961-145d-4e9d-b66e-a0bc06cc6065" path="/var/lib/kubelet/pods/86c42961-145d-4e9d-b66e-a0bc06cc6065/volumes" Nov 03 02:00:20.092179 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:20.092162 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb08db38-b50e-4488-ada1-ffce3cf76fd1" path="/var/lib/kubelet/pods/cb08db38-b50e-4488-ada1-ffce3cf76fd1/volumes" Nov 03 02:00:21.926869 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:21.926831 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:00:24.775908 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:24.775856 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:00:31.135730 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:31.135696 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:00:31.389957 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:31.389868 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:00:31.594346 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:31.594311 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:00:36.109943 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:36.109905 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:00:41.043093 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:41.043051 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:00:44.037879 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:00:44.037843 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 02:00:45.744708 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:45.744671 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:00:46.320939 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:46.320902 5437 generic.go:356] "Generic (PLEG): container finished" podID="1ce8bfdb-a81b-4641-996a-c5fe222b1e8c" containerID="d46029c563cc8d26b81544758389e98c18dbc749005e5d7ec5a52db5cd033d08" exitCode=0 Nov 03 02:00:46.321128 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:46.320957 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-e0907d-openshift-default-59786747c7-479hs" event={"ID":"1ce8bfdb-a81b-4641-996a-c5fe222b1e8c","Type":"ContainerDied","Data":"d46029c563cc8d26b81544758389e98c18dbc749005e5d7ec5a52db5cd033d08"} Nov 03 02:00:46.321391 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:46.321351 5437 scope.go:122] "RemoveContainer" containerID="d46029c563cc8d26b81544758389e98c18dbc749005e5d7ec5a52db5cd033d08" Nov 03 02:00:46.322915 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:46.322885 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 02:00:46.323033 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:46.322957 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 02:00:46.323033 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:46.322992 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 02:00:47.349334 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:47.349243 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-e0907d-openshift-default-59786747c7-479hs" event={"ID":"1ce8bfdb-a81b-4641-996a-c5fe222b1e8c","Type":"ContainerStarted","Data":"0027529b6afedab89964a2812a63b6aad847fe93416d6ca4dd91a75a4c14bc54"} Nov 03 02:00:47.445396 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:47.436729 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-e0907d-openshift-default-59786747c7-479hs" Nov 03 02:00:47.445396 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:47.436876 5437 patch_prober.go:29] interesting pod/e2e-praxis-b-e0907d-openshift-default-59786747c7-479hs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Nov 03 02:00:47.445396 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:47.436922 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-e0907d-openshift-default-59786747c7-479hs" podUID="1ce8bfdb-a81b-4641-996a-c5fe222b1e8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Nov 03 02:00:47.588508 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:47.588459 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-e0907d-openshift-default-59786747c7-479hs" Nov 03 02:00:48.434804 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:48.434763 5437 patch_prober.go:29] interesting pod/e2e-praxis-b-e0907d-openshift-default-59786747c7-479hs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Nov 03 02:00:48.435445 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:48.435410 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-e0907d-openshift-default-59786747c7-479hs" podUID="1ce8bfdb-a81b-4641-996a-c5fe222b1e8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Nov 03 02:00:49.434825 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:49.434781 5437 patch_prober.go:29] interesting pod/e2e-praxis-b-e0907d-openshift-default-59786747c7-479hs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Nov 03 02:00:49.435480 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:49.435446 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-e0907d-openshift-default-59786747c7-479hs" podUID="1ce8bfdb-a81b-4641-996a-c5fe222b1e8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Nov 03 02:00:50.443451 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:50.443398 5437 patch_prober.go:29] interesting pod/e2e-praxis-b-e0907d-openshift-default-59786747c7-479hs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Nov 03 02:00:50.443977 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:50.443477 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-e0907d-openshift-default-59786747c7-479hs" podUID="1ce8bfdb-a81b-4641-996a-c5fe222b1e8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Nov 03 02:00:50.467998 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:50.467963 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:00:50.707909 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:50.707820 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:00:51.439010 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:51.438968 5437 patch_prober.go:29] interesting pod/e2e-praxis-b-e0907d-openshift-default-59786747c7-479hs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Nov 03 02:00:51.439316 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:51.439275 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-e0907d-openshift-default-59786747c7-479hs" podUID="1ce8bfdb-a81b-4641-996a-c5fe222b1e8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Nov 03 02:00:52.433860 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:52.433821 5437 patch_prober.go:29] interesting pod/e2e-praxis-b-e0907d-openshift-default-59786747c7-479hs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Nov 03 02:00:52.434426 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:52.433902 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-e0907d-openshift-default-59786747c7-479hs" podUID="1ce8bfdb-a81b-4641-996a-c5fe222b1e8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Nov 03 02:00:53.439950 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:53.439914 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-e0907d-openshift-default-59786747c7-479hs" Nov 03 02:00:53.441177 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:53.441152 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-e0907d-openshift-default-59786747c7-479hs" Nov 03 02:00:54.781111 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:54.781074 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:00:55.466248 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:55.466211 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:00:57.352044 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:57.352009 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n8wz8"] Nov 03 02:00:57.352665 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:57.352225 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n8wz8" podUID="9b955a41-8e08-46b0-87d0-64ab0cc1e381" containerName="istio-proxy" containerID="cri-o://3511ea5db74e3c69162bea6aeac9babd9d71c73b9d9d163be8902526583026c4" gracePeriod=30 Nov 03 02:00:59.627327 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:00:59.627289 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:01:00.726215 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:00.726174 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:01:01.212093 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:01.212048 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:01:01.415178 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:01.415143 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:01:01.522941 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:01.522515 5437 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-n8wz8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 02:01:01.522941 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:01.522589 5437 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n8wz8" podUID="9b955a41-8e08-46b0-87d0-64ab0cc1e381" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 02:01:01.873228 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:01.873185 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:01:02.378284 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:02.378253 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:01:02.638707 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:02.638633 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n8wz8" Nov 03 02:01:02.740654 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:02.740617 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b955a41-8e08-46b0-87d0-64ab0cc1e381-kube-api-access-6hfxj\" (UniqueName: \"kubernetes.io/projected/9b955a41-8e08-46b0-87d0-64ab0cc1e381-kube-api-access-6hfxj\") pod \"9b955a41-8e08-46b0-87d0-64ab0cc1e381\" (UID: \"9b955a41-8e08-46b0-87d0-64ab0cc1e381\") " Nov 03 02:01:02.740870 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:02.740673 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b955a41-8e08-46b0-87d0-64ab0cc1e381-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b955a41-8e08-46b0-87d0-64ab0cc1e381-workload-certs\") pod \"9b955a41-8e08-46b0-87d0-64ab0cc1e381\" (UID: \"9b955a41-8e08-46b0-87d0-64ab0cc1e381\") " Nov 03 02:01:02.740870 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:02.740757 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b955a41-8e08-46b0-87d0-64ab0cc1e381-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b955a41-8e08-46b0-87d0-64ab0cc1e381-istio-data\") pod \"9b955a41-8e08-46b0-87d0-64ab0cc1e381\" (UID: \"9b955a41-8e08-46b0-87d0-64ab0cc1e381\") " Nov 03 02:01:02.740870 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:02.740804 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b955a41-8e08-46b0-87d0-64ab0cc1e381-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b955a41-8e08-46b0-87d0-64ab0cc1e381-credential-socket\") pod \"9b955a41-8e08-46b0-87d0-64ab0cc1e381\" (UID: \"9b955a41-8e08-46b0-87d0-64ab0cc1e381\") " Nov 03 02:01:02.741057 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:02.740914 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b955a41-8e08-46b0-87d0-64ab0cc1e381-istio-token\" (UniqueName: \"kubernetes.io/projected/9b955a41-8e08-46b0-87d0-64ab0cc1e381-istio-token\") pod \"9b955a41-8e08-46b0-87d0-64ab0cc1e381\" (UID: \"9b955a41-8e08-46b0-87d0-64ab0cc1e381\") " Nov 03 02:01:02.741057 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:02.740951 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b955a41-8e08-46b0-87d0-64ab0cc1e381-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b955a41-8e08-46b0-87d0-64ab0cc1e381-istio-envoy\") pod \"9b955a41-8e08-46b0-87d0-64ab0cc1e381\" (UID: \"9b955a41-8e08-46b0-87d0-64ab0cc1e381\") " Nov 03 02:01:02.741057 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:02.741000 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b955a41-8e08-46b0-87d0-64ab0cc1e381-workload-certs" pod "9b955a41-8e08-46b0-87d0-64ab0cc1e381" (UID: "9b955a41-8e08-46b0-87d0-64ab0cc1e381"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:01:02.741057 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:02.741028 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9b955a41-8e08-46b0-87d0-64ab0cc1e381-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b955a41-8e08-46b0-87d0-64ab0cc1e381-istio-podinfo\") pod \"9b955a41-8e08-46b0-87d0-64ab0cc1e381\" (UID: \"9b955a41-8e08-46b0-87d0-64ab0cc1e381\") " Nov 03 02:01:02.741276 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:02.741108 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b955a41-8e08-46b0-87d0-64ab0cc1e381-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b955a41-8e08-46b0-87d0-64ab0cc1e381-workload-socket\") pod \"9b955a41-8e08-46b0-87d0-64ab0cc1e381\" (UID: \"9b955a41-8e08-46b0-87d0-64ab0cc1e381\") " Nov 03 02:01:02.741276 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:02.741137 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b955a41-8e08-46b0-87d0-64ab0cc1e381-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b955a41-8e08-46b0-87d0-64ab0cc1e381-istiod-ca-cert\") pod \"9b955a41-8e08-46b0-87d0-64ab0cc1e381\" (UID: \"9b955a41-8e08-46b0-87d0-64ab0cc1e381\") " Nov 03 02:01:02.741440 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:02.741138 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b955a41-8e08-46b0-87d0-64ab0cc1e381-istio-data" pod "9b955a41-8e08-46b0-87d0-64ab0cc1e381" (UID: "9b955a41-8e08-46b0-87d0-64ab0cc1e381"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:01:02.741440 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:02.741252 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b955a41-8e08-46b0-87d0-64ab0cc1e381-credential-socket" pod "9b955a41-8e08-46b0-87d0-64ab0cc1e381" (UID: "9b955a41-8e08-46b0-87d0-64ab0cc1e381"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:01:02.741653 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:02.741479 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b955a41-8e08-46b0-87d0-64ab0cc1e381-workload-socket" pod "9b955a41-8e08-46b0-87d0-64ab0cc1e381" (UID: "9b955a41-8e08-46b0-87d0-64ab0cc1e381"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:01:02.741757 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:02.741735 5437 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b955a41-8e08-46b0-87d0-64ab0cc1e381-workload-socket\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:01:02.741823 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:02.741761 5437 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b955a41-8e08-46b0-87d0-64ab0cc1e381-workload-certs\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:01:02.741823 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:02.741777 5437 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b955a41-8e08-46b0-87d0-64ab0cc1e381-istio-data\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:01:02.741823 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:02.741791 5437 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b955a41-8e08-46b0-87d0-64ab0cc1e381-credential-socket\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:01:02.742313 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:02.742269 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b955a41-8e08-46b0-87d0-64ab0cc1e381-istiod-ca-cert" pod "9b955a41-8e08-46b0-87d0-64ab0cc1e381" (UID: "9b955a41-8e08-46b0-87d0-64ab0cc1e381"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:01:02.743671 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:02.743641 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b955a41-8e08-46b0-87d0-64ab0cc1e381-kube-api-access-6hfxj" pod "9b955a41-8e08-46b0-87d0-64ab0cc1e381" (UID: "9b955a41-8e08-46b0-87d0-64ab0cc1e381"). InnerVolumeSpecName "kube-api-access-6hfxj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:01:02.743947 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:02.743910 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b955a41-8e08-46b0-87d0-64ab0cc1e381-istio-token" pod "9b955a41-8e08-46b0-87d0-64ab0cc1e381" (UID: "9b955a41-8e08-46b0-87d0-64ab0cc1e381"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:01:02.744066 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:02.744037 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b955a41-8e08-46b0-87d0-64ab0cc1e381-istio-envoy" pod "9b955a41-8e08-46b0-87d0-64ab0cc1e381" (UID: "9b955a41-8e08-46b0-87d0-64ab0cc1e381"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:01:02.744263 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:02.744238 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9b955a41-8e08-46b0-87d0-64ab0cc1e381-istio-podinfo" pod "9b955a41-8e08-46b0-87d0-64ab0cc1e381" (UID: "9b955a41-8e08-46b0-87d0-64ab0cc1e381"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:01:02.843237 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:02.843192 5437 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b955a41-8e08-46b0-87d0-64ab0cc1e381-istio-token\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:01:02.843237 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:02.843224 5437 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b955a41-8e08-46b0-87d0-64ab0cc1e381-istio-envoy\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:01:02.843237 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:02.843237 5437 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b955a41-8e08-46b0-87d0-64ab0cc1e381-istio-podinfo\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:01:02.843598 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:02.843250 5437 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b955a41-8e08-46b0-87d0-64ab0cc1e381-istiod-ca-cert\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:01:02.843598 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:02.843262 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hfxj\" (UniqueName: \"kubernetes.io/projected/9b955a41-8e08-46b0-87d0-64ab0cc1e381-kube-api-access-6hfxj\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:01:02.896182 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:02.896093 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:01:03.104917 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:03.104875 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:01:03.457582 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:03.457546 5437 generic.go:356] "Generic (PLEG): container finished" podID="9b955a41-8e08-46b0-87d0-64ab0cc1e381" containerID="3511ea5db74e3c69162bea6aeac9babd9d71c73b9d9d163be8902526583026c4" exitCode=0 Nov 03 02:01:03.457789 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:03.457597 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n8wz8" Nov 03 02:01:03.457789 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:03.457626 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n8wz8" event={"ID":"9b955a41-8e08-46b0-87d0-64ab0cc1e381","Type":"ContainerDied","Data":"3511ea5db74e3c69162bea6aeac9babd9d71c73b9d9d163be8902526583026c4"} Nov 03 02:01:03.457789 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:03.457682 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n8wz8" event={"ID":"9b955a41-8e08-46b0-87d0-64ab0cc1e381","Type":"ContainerDied","Data":"b8e87c34169361b3c79991b7273a47c0e4870299fd73ec95cd50d555d3cc9d51"} Nov 03 02:01:03.457789 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:03.457704 5437 scope.go:122] "RemoveContainer" containerID="3511ea5db74e3c69162bea6aeac9babd9d71c73b9d9d163be8902526583026c4" Nov 03 02:01:03.469299 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:03.469277 5437 scope.go:122] "RemoveContainer" containerID="3511ea5db74e3c69162bea6aeac9babd9d71c73b9d9d163be8902526583026c4" Nov 03 02:01:03.469695 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:01:03.469669 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3511ea5db74e3c69162bea6aeac9babd9d71c73b9d9d163be8902526583026c4\": container with ID starting with 3511ea5db74e3c69162bea6aeac9babd9d71c73b9d9d163be8902526583026c4 not found: ID does not exist" containerID="3511ea5db74e3c69162bea6aeac9babd9d71c73b9d9d163be8902526583026c4" Nov 03 02:01:03.469745 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:03.469706 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3511ea5db74e3c69162bea6aeac9babd9d71c73b9d9d163be8902526583026c4"} err="failed to get container status \"3511ea5db74e3c69162bea6aeac9babd9d71c73b9d9d163be8902526583026c4\": rpc error: code = NotFound desc = could not find container \"3511ea5db74e3c69162bea6aeac9babd9d71c73b9d9d163be8902526583026c4\": container with ID starting with 3511ea5db74e3c69162bea6aeac9babd9d71c73b9d9d163be8902526583026c4 not found: ID does not exist" Nov 03 02:01:03.682388 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:03.682341 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n8wz8"] Nov 03 02:01:03.735668 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:03.735568 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-n8wz8"] Nov 03 02:01:03.809948 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:03.809914 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:01:04.008901 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:04.008817 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:01:04.088719 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:04.088687 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b955a41-8e08-46b0-87d0-64ab0cc1e381" path="/var/lib/kubelet/pods/9b955a41-8e08-46b0-87d0-64ab0cc1e381/volumes" Nov 03 02:01:05.276137 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:05.276104 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:01:05.808267 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:05.808215 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:01:06.485137 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:06.485082 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:01:06.691910 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:06.691867 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:01:08.005270 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:08.005227 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:01:08.503173 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:08.503130 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:01:08.969975 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:08.969939 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:01:09.171760 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:09.171720 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:01:12.765302 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:12.765259 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:01:13.072986 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:13.072944 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-a00717-84ffc588b5-k8cfr"] Nov 03 02:01:13.181065 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:13.181032 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-a00717-76799c76dc-7jl9f"] Nov 03 02:01:13.482236 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:13.482135 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:01:13.687532 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:13.687493 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:01:14.034561 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:01:14.034520 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 02:01:14.430770 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:14.430717 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:01:15.153862 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:15.153823 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:01:18.073863 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:18.073816 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-a00717-84ffc588b5-k8cfr" podUID="775139d4-2b62-484e-8d22-ae25e9e2d4d5" containerName="payload-processing" containerID="cri-o://12bdbdf61526e81cec961a8dfd57a6d64981c917ca4d8d3ec4e667cf374648c3" gracePeriod=25 Nov 03 02:01:18.181905 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:18.181826 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a00717-76799c76dc-7jl9f" podUID="a255ffd8-6415-41b7-af49-0e6522ff1133" containerName="payload-pre-processing" containerID="cri-o://c5c37b40b4373fe19616ff677acb2e3fcaff97f899fa6d1901994f7e685db86d" gracePeriod=25 Nov 03 02:01:18.343485 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:18.343409 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-a00717-84ffc588b5-k8cfr" Nov 03 02:01:18.444109 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:18.444067 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/775139d4-2b62-484e-8d22-ae25e9e2d4d5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/775139d4-2b62-484e-8d22-ae25e9e2d4d5-plugins-config-volume\") pod \"775139d4-2b62-484e-8d22-ae25e9e2d4d5\" (UID: \"775139d4-2b62-484e-8d22-ae25e9e2d4d5\") " Nov 03 02:01:18.444789 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:18.444279 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/775139d4-2b62-484e-8d22-ae25e9e2d4d5-kube-api-access-j5bhw\" (UniqueName: \"kubernetes.io/projected/775139d4-2b62-484e-8d22-ae25e9e2d4d5-kube-api-access-j5bhw\") pod \"775139d4-2b62-484e-8d22-ae25e9e2d4d5\" (UID: \"775139d4-2b62-484e-8d22-ae25e9e2d4d5\") " Nov 03 02:01:18.444789 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:18.444525 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/775139d4-2b62-484e-8d22-ae25e9e2d4d5-plugins-config-volume" pod "775139d4-2b62-484e-8d22-ae25e9e2d4d5" (UID: "775139d4-2b62-484e-8d22-ae25e9e2d4d5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:01:18.444789 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:18.444716 5437 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/775139d4-2b62-484e-8d22-ae25e9e2d4d5-plugins-config-volume\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:01:18.446574 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:18.446550 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/775139d4-2b62-484e-8d22-ae25e9e2d4d5-kube-api-access-j5bhw" pod "775139d4-2b62-484e-8d22-ae25e9e2d4d5" (UID: "775139d4-2b62-484e-8d22-ae25e9e2d4d5"). InnerVolumeSpecName "kube-api-access-j5bhw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:01:18.458786 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:18.458693 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a00717-76799c76dc-7jl9f" Nov 03 02:01:18.530578 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:18.530532 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:01:18.546043 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:18.546015 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a255ffd8-6415-41b7-af49-0e6522ff1133-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a255ffd8-6415-41b7-af49-0e6522ff1133-plugins-config-volume\") pod \"a255ffd8-6415-41b7-af49-0e6522ff1133\" (UID: \"a255ffd8-6415-41b7-af49-0e6522ff1133\") " Nov 03 02:01:18.546238 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:18.546151 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a255ffd8-6415-41b7-af49-0e6522ff1133-kube-api-access-49287\" (UniqueName: \"kubernetes.io/projected/a255ffd8-6415-41b7-af49-0e6522ff1133-kube-api-access-49287\") pod \"a255ffd8-6415-41b7-af49-0e6522ff1133\" (UID: \"a255ffd8-6415-41b7-af49-0e6522ff1133\") " Nov 03 02:01:18.546434 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:18.546418 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5bhw\" (UniqueName: \"kubernetes.io/projected/775139d4-2b62-484e-8d22-ae25e9e2d4d5-kube-api-access-j5bhw\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:01:18.546505 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:18.546447 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a255ffd8-6415-41b7-af49-0e6522ff1133-plugins-config-volume" pod "a255ffd8-6415-41b7-af49-0e6522ff1133" (UID: "a255ffd8-6415-41b7-af49-0e6522ff1133"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:01:18.548199 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:18.548167 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a255ffd8-6415-41b7-af49-0e6522ff1133-kube-api-access-49287" pod "a255ffd8-6415-41b7-af49-0e6522ff1133" (UID: "a255ffd8-6415-41b7-af49-0e6522ff1133"). InnerVolumeSpecName "kube-api-access-49287". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:01:18.554406 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:18.554380 5437 generic.go:356] "Generic (PLEG): container finished" podID="a255ffd8-6415-41b7-af49-0e6522ff1133" containerID="c5c37b40b4373fe19616ff677acb2e3fcaff97f899fa6d1901994f7e685db86d" exitCode=0 Nov 03 02:01:18.554520 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:18.554436 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a00717-76799c76dc-7jl9f" Nov 03 02:01:18.554520 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:18.554451 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a00717-76799c76dc-7jl9f" event={"ID":"a255ffd8-6415-41b7-af49-0e6522ff1133","Type":"ContainerDied","Data":"c5c37b40b4373fe19616ff677acb2e3fcaff97f899fa6d1901994f7e685db86d"} Nov 03 02:01:18.554520 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:18.554505 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a00717-76799c76dc-7jl9f" event={"ID":"a255ffd8-6415-41b7-af49-0e6522ff1133","Type":"ContainerDied","Data":"8806c3255933b86832a6761373c368f9379618b05c2e43af061761c0fb6c76d8"} Nov 03 02:01:18.554655 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:18.554539 5437 scope.go:122] "RemoveContainer" containerID="c5c37b40b4373fe19616ff677acb2e3fcaff97f899fa6d1901994f7e685db86d" Nov 03 02:01:18.556057 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:18.556034 5437 generic.go:356] "Generic (PLEG): container finished" podID="775139d4-2b62-484e-8d22-ae25e9e2d4d5" containerID="12bdbdf61526e81cec961a8dfd57a6d64981c917ca4d8d3ec4e667cf374648c3" exitCode=0 Nov 03 02:01:18.556160 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:18.556079 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-a00717-84ffc588b5-k8cfr" Nov 03 02:01:18.556160 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:18.556090 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-a00717-84ffc588b5-k8cfr" event={"ID":"775139d4-2b62-484e-8d22-ae25e9e2d4d5","Type":"ContainerDied","Data":"12bdbdf61526e81cec961a8dfd57a6d64981c917ca4d8d3ec4e667cf374648c3"} Nov 03 02:01:18.556160 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:18.556140 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-a00717-84ffc588b5-k8cfr" event={"ID":"775139d4-2b62-484e-8d22-ae25e9e2d4d5","Type":"ContainerDied","Data":"74d95f7402fc788181645f8a20a946c03afaf943eed3abc628a27be41a47ee99"} Nov 03 02:01:18.564600 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:18.564581 5437 scope.go:122] "RemoveContainer" containerID="c5c37b40b4373fe19616ff677acb2e3fcaff97f899fa6d1901994f7e685db86d" Nov 03 02:01:18.564950 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:01:18.564925 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5c37b40b4373fe19616ff677acb2e3fcaff97f899fa6d1901994f7e685db86d\": container with ID starting with c5c37b40b4373fe19616ff677acb2e3fcaff97f899fa6d1901994f7e685db86d not found: ID does not exist" containerID="c5c37b40b4373fe19616ff677acb2e3fcaff97f899fa6d1901994f7e685db86d" Nov 03 02:01:18.565060 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:18.564962 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5c37b40b4373fe19616ff677acb2e3fcaff97f899fa6d1901994f7e685db86d"} err="failed to get container status \"c5c37b40b4373fe19616ff677acb2e3fcaff97f899fa6d1901994f7e685db86d\": rpc error: code = NotFound desc = could not find container \"c5c37b40b4373fe19616ff677acb2e3fcaff97f899fa6d1901994f7e685db86d\": container with ID starting with c5c37b40b4373fe19616ff677acb2e3fcaff97f899fa6d1901994f7e685db86d not found: ID does not exist" Nov 03 02:01:18.565060 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:18.564986 5437 scope.go:122] "RemoveContainer" containerID="12bdbdf61526e81cec961a8dfd57a6d64981c917ca4d8d3ec4e667cf374648c3" Nov 03 02:01:18.574519 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:18.574500 5437 scope.go:122] "RemoveContainer" containerID="12bdbdf61526e81cec961a8dfd57a6d64981c917ca4d8d3ec4e667cf374648c3" Nov 03 02:01:18.574773 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:01:18.574753 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12bdbdf61526e81cec961a8dfd57a6d64981c917ca4d8d3ec4e667cf374648c3\": container with ID starting with 12bdbdf61526e81cec961a8dfd57a6d64981c917ca4d8d3ec4e667cf374648c3 not found: ID does not exist" containerID="12bdbdf61526e81cec961a8dfd57a6d64981c917ca4d8d3ec4e667cf374648c3" Nov 03 02:01:18.574827 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:18.574781 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12bdbdf61526e81cec961a8dfd57a6d64981c917ca4d8d3ec4e667cf374648c3"} err="failed to get container status \"12bdbdf61526e81cec961a8dfd57a6d64981c917ca4d8d3ec4e667cf374648c3\": rpc error: code = NotFound desc = could not find container \"12bdbdf61526e81cec961a8dfd57a6d64981c917ca4d8d3ec4e667cf374648c3\": container with ID starting with 12bdbdf61526e81cec961a8dfd57a6d64981c917ca4d8d3ec4e667cf374648c3 not found: ID does not exist" Nov 03 02:01:18.647247 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:18.647146 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49287\" (UniqueName: \"kubernetes.io/projected/a255ffd8-6415-41b7-af49-0e6522ff1133-kube-api-access-49287\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:01:18.647247 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:18.647183 5437 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a255ffd8-6415-41b7-af49-0e6522ff1133-plugins-config-volume\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:01:18.699215 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:18.699175 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-a00717-84ffc588b5-k8cfr"] Nov 03 02:01:18.703002 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:18.702969 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-a00717-84ffc588b5-k8cfr"] Nov 03 02:01:18.816072 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:18.816036 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-a00717-76799c76dc-7jl9f"] Nov 03 02:01:18.819741 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:18.819715 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-a00717-76799c76dc-7jl9f"] Nov 03 02:01:20.090032 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:20.089998 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="775139d4-2b62-484e-8d22-ae25e9e2d4d5" path="/var/lib/kubelet/pods/775139d4-2b62-484e-8d22-ae25e9e2d4d5/volumes" Nov 03 02:01:20.090456 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:20.090429 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a255ffd8-6415-41b7-af49-0e6522ff1133" path="/var/lib/kubelet/pods/a255ffd8-6415-41b7-af49-0e6522ff1133/volumes" Nov 03 02:01:21.197569 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:21.197504 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:01:26.063565 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:26.063524 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:01:30.081026 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:30.080978 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-738acb-6c99bd8c8f-8zf4p"] Nov 03 02:01:30.081565 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:30.081233 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-738acb-6c99bd8c8f-8zf4p" podUID="306b59aa-822c-4e6c-9434-bc06393c2802" containerName="maas-api" containerID="cri-o://46c0256b12beeb4f44be6462fe25c3e4c6cd2761bf893e938eed6430ccf95d1f" gracePeriod=30 Nov 03 02:01:30.238038 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:30.237994 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:01:30.373087 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:30.373056 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-738acb-6c99bd8c8f-8zf4p" Nov 03 02:01:30.494166 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:30.494124 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/306b59aa-822c-4e6c-9434-bc06393c2802-metrics-certs\" (UniqueName: \"kubernetes.io/secret/306b59aa-822c-4e6c-9434-bc06393c2802-metrics-certs\") pod \"306b59aa-822c-4e6c-9434-bc06393c2802\" (UID: \"306b59aa-822c-4e6c-9434-bc06393c2802\") " Nov 03 02:01:30.494355 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:30.494196 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/306b59aa-822c-4e6c-9434-bc06393c2802-kube-api-access-9vvxs\" (UniqueName: \"kubernetes.io/projected/306b59aa-822c-4e6c-9434-bc06393c2802-kube-api-access-9vvxs\") pod \"306b59aa-822c-4e6c-9434-bc06393c2802\" (UID: \"306b59aa-822c-4e6c-9434-bc06393c2802\") " Nov 03 02:01:30.494468 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:30.494357 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/306b59aa-822c-4e6c-9434-bc06393c2802-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/306b59aa-822c-4e6c-9434-bc06393c2802-openshift-service-ca\") pod \"306b59aa-822c-4e6c-9434-bc06393c2802\" (UID: \"306b59aa-822c-4e6c-9434-bc06393c2802\") " Nov 03 02:01:30.494468 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:30.494436 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/306b59aa-822c-4e6c-9434-bc06393c2802-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/306b59aa-822c-4e6c-9434-bc06393c2802-maas-api-tls\") pod \"306b59aa-822c-4e6c-9434-bc06393c2802\" (UID: \"306b59aa-822c-4e6c-9434-bc06393c2802\") " Nov 03 02:01:30.494986 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:30.494959 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/306b59aa-822c-4e6c-9434-bc06393c2802-openshift-service-ca" pod "306b59aa-822c-4e6c-9434-bc06393c2802" (UID: "306b59aa-822c-4e6c-9434-bc06393c2802"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:01:30.497721 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:30.497677 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/306b59aa-822c-4e6c-9434-bc06393c2802-kube-api-access-9vvxs" pod "306b59aa-822c-4e6c-9434-bc06393c2802" (UID: "306b59aa-822c-4e6c-9434-bc06393c2802"). InnerVolumeSpecName "kube-api-access-9vvxs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:01:30.498051 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:30.498020 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/306b59aa-822c-4e6c-9434-bc06393c2802-maas-api-tls" pod "306b59aa-822c-4e6c-9434-bc06393c2802" (UID: "306b59aa-822c-4e6c-9434-bc06393c2802"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:01:30.498188 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:30.498153 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/306b59aa-822c-4e6c-9434-bc06393c2802-metrics-certs" pod "306b59aa-822c-4e6c-9434-bc06393c2802" (UID: "306b59aa-822c-4e6c-9434-bc06393c2802"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:01:30.596188 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:30.596096 5437 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/306b59aa-822c-4e6c-9434-bc06393c2802-openshift-service-ca\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:01:30.596188 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:30.596128 5437 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/306b59aa-822c-4e6c-9434-bc06393c2802-maas-api-tls\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:01:30.596188 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:30.596136 5437 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/306b59aa-822c-4e6c-9434-bc06393c2802-metrics-certs\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:01:30.596188 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:30.596145 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vvxs\" (UniqueName: \"kubernetes.io/projected/306b59aa-822c-4e6c-9434-bc06393c2802-kube-api-access-9vvxs\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:01:30.635084 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:30.635042 5437 generic.go:356] "Generic (PLEG): container finished" podID="306b59aa-822c-4e6c-9434-bc06393c2802" containerID="46c0256b12beeb4f44be6462fe25c3e4c6cd2761bf893e938eed6430ccf95d1f" exitCode=0 Nov 03 02:01:30.635281 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:30.635097 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-738acb-6c99bd8c8f-8zf4p" Nov 03 02:01:30.635281 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:30.635111 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-738acb-6c99bd8c8f-8zf4p" event={"ID":"306b59aa-822c-4e6c-9434-bc06393c2802","Type":"ContainerDied","Data":"46c0256b12beeb4f44be6462fe25c3e4c6cd2761bf893e938eed6430ccf95d1f"} Nov 03 02:01:30.635281 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:30.635186 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-738acb-6c99bd8c8f-8zf4p" event={"ID":"306b59aa-822c-4e6c-9434-bc06393c2802","Type":"ContainerDied","Data":"3fe06aab1a69c97b37de3e30e604253cc892b1c87a6fa3b95e8efa219c596283"} Nov 03 02:01:30.635281 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:30.635221 5437 scope.go:122] "RemoveContainer" containerID="46c0256b12beeb4f44be6462fe25c3e4c6cd2761bf893e938eed6430ccf95d1f" Nov 03 02:01:30.646877 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:30.646850 5437 scope.go:122] "RemoveContainer" containerID="46c0256b12beeb4f44be6462fe25c3e4c6cd2761bf893e938eed6430ccf95d1f" Nov 03 02:01:30.647249 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:01:30.647220 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46c0256b12beeb4f44be6462fe25c3e4c6cd2761bf893e938eed6430ccf95d1f\": container with ID starting with 46c0256b12beeb4f44be6462fe25c3e4c6cd2761bf893e938eed6430ccf95d1f not found: ID does not exist" containerID="46c0256b12beeb4f44be6462fe25c3e4c6cd2761bf893e938eed6430ccf95d1f" Nov 03 02:01:30.647304 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:30.647266 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46c0256b12beeb4f44be6462fe25c3e4c6cd2761bf893e938eed6430ccf95d1f"} err="failed to get container status \"46c0256b12beeb4f44be6462fe25c3e4c6cd2761bf893e938eed6430ccf95d1f\": rpc error: code = NotFound desc = could not find container \"46c0256b12beeb4f44be6462fe25c3e4c6cd2761bf893e938eed6430ccf95d1f\": container with ID starting with 46c0256b12beeb4f44be6462fe25c3e4c6cd2761bf893e938eed6430ccf95d1f not found: ID does not exist" Nov 03 02:01:30.739416 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:30.739375 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:01:30.806201 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:30.806162 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-738acb-6c99bd8c8f-8zf4p"] Nov 03 02:01:30.812503 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:30.812474 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-738acb-6c99bd8c8f-8zf4p"] Nov 03 02:01:30.946993 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:30.946903 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:01:32.081282 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:32.081238 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:01:32.088754 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:32.088720 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="306b59aa-822c-4e6c-9434-bc06393c2802" path="/var/lib/kubelet/pods/306b59aa-822c-4e6c-9434-bc06393c2802/volumes" Nov 03 02:01:33.056540 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:33.056501 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:01:35.978753 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:35.978711 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:01:37.218509 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:37.218338 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:01:38.852242 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:38.852204 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:01:39.055067 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:39.055030 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:01:39.723994 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:39.723944 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:01:40.449773 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:40.449725 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:01:41.000339 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:41.000292 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-738acb-openshift-default-c7bc4fcf7-llk59"] Nov 03 02:01:41.000559 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:41.000532 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-738acb-openshift-default-c7bc4fcf7-llk59" podUID="a06e089e-fba5-403f-9fe5-34175c6865fa" containerName="istio-proxy" containerID="cri-o://fdded550c5957428c7f6d533d35877beeabf2bd33a185dc03a2ae3c1c1d821b2" gracePeriod=30 Nov 03 02:01:41.062956 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:41.062913 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:01:41.873730 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:41.873665 5437 patch_prober.go:29] interesting pod/e2e-ipp-b-738acb-openshift-default-c7bc4fcf7-llk59 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 02:01:41.874238 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:41.873764 5437 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-738acb-openshift-default-c7bc4fcf7-llk59" podUID="a06e089e-fba5-403f-9fe5-34175c6865fa" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 02:01:42.058571 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:42.058535 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:01:42.355837 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:42.355795 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:01:42.615453 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:42.615334 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-70235a-6f9d5df8dc-xgzc9"] Nov 03 02:01:42.718544 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:42.718505 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-70235a-74f5459759-7hw6j"] Nov 03 02:01:43.034474 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:43.034312 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:01:43.239179 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:43.239146 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:01:43.489817 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:43.489786 5437 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-ghxht_c4e1a934-d917-417b-aea3-d432d2cff990/payload-processing/0.log" Nov 03 02:01:43.741775 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:43.741688 5437 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-ghxht_c4e1a934-d917-417b-aea3-d432d2cff990/payload-processing/0.log" Nov 03 02:01:43.977089 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:43.977062 5437 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-db583d-7cf47dd48c-lrzwz_768f8efe-c962-4d38-8743-a835df279bd7/payload-processing/0.log" Nov 03 02:01:44.036795 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:01:44.036701 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 02:01:44.217197 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:44.217170 5437 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-db583d-7cf47dd48c-lrzwz_768f8efe-c962-4d38-8743-a835df279bd7/payload-processing/0.log" Nov 03 02:01:44.895411 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:44.895357 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-db583d-7cf47dd48c-lrzwz"] Nov 03 02:01:45.301349 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:45.301312 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:01:45.501804 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:45.501769 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:01:46.280094 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:46.280001 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-738acb-openshift-default-c7bc4fcf7-llk59" Nov 03 02:01:46.376293 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:46.376232 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a06e089e-fba5-403f-9fe5-34175c6865fa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a06e089e-fba5-403f-9fe5-34175c6865fa-istio-data\") pod \"a06e089e-fba5-403f-9fe5-34175c6865fa\" (UID: \"a06e089e-fba5-403f-9fe5-34175c6865fa\") " Nov 03 02:01:46.376293 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:46.376309 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a06e089e-fba5-403f-9fe5-34175c6865fa-kube-api-access-t7l5x\" (UniqueName: \"kubernetes.io/projected/a06e089e-fba5-403f-9fe5-34175c6865fa-kube-api-access-t7l5x\") pod \"a06e089e-fba5-403f-9fe5-34175c6865fa\" (UID: \"a06e089e-fba5-403f-9fe5-34175c6865fa\") " Nov 03 02:01:46.376904 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:46.376342 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a06e089e-fba5-403f-9fe5-34175c6865fa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a06e089e-fba5-403f-9fe5-34175c6865fa-workload-certs\") pod \"a06e089e-fba5-403f-9fe5-34175c6865fa\" (UID: \"a06e089e-fba5-403f-9fe5-34175c6865fa\") " Nov 03 02:01:46.376904 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:46.376402 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a06e089e-fba5-403f-9fe5-34175c6865fa-istio-token\" (UniqueName: \"kubernetes.io/projected/a06e089e-fba5-403f-9fe5-34175c6865fa-istio-token\") pod \"a06e089e-fba5-403f-9fe5-34175c6865fa\" (UID: \"a06e089e-fba5-403f-9fe5-34175c6865fa\") " Nov 03 02:01:46.376904 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:46.376445 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a06e089e-fba5-403f-9fe5-34175c6865fa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a06e089e-fba5-403f-9fe5-34175c6865fa-credential-socket\") pod \"a06e089e-fba5-403f-9fe5-34175c6865fa\" (UID: \"a06e089e-fba5-403f-9fe5-34175c6865fa\") " Nov 03 02:01:46.376904 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:46.376480 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a06e089e-fba5-403f-9fe5-34175c6865fa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a06e089e-fba5-403f-9fe5-34175c6865fa-istiod-ca-cert\") pod \"a06e089e-fba5-403f-9fe5-34175c6865fa\" (UID: \"a06e089e-fba5-403f-9fe5-34175c6865fa\") " Nov 03 02:01:46.376904 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:46.376505 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a06e089e-fba5-403f-9fe5-34175c6865fa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a06e089e-fba5-403f-9fe5-34175c6865fa-istio-podinfo\") pod \"a06e089e-fba5-403f-9fe5-34175c6865fa\" (UID: \"a06e089e-fba5-403f-9fe5-34175c6865fa\") " Nov 03 02:01:46.376904 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:46.376560 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a06e089e-fba5-403f-9fe5-34175c6865fa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a06e089e-fba5-403f-9fe5-34175c6865fa-workload-socket\") pod \"a06e089e-fba5-403f-9fe5-34175c6865fa\" (UID: \"a06e089e-fba5-403f-9fe5-34175c6865fa\") " Nov 03 02:01:46.376904 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:46.376583 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a06e089e-fba5-403f-9fe5-34175c6865fa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a06e089e-fba5-403f-9fe5-34175c6865fa-istio-envoy\") pod \"a06e089e-fba5-403f-9fe5-34175c6865fa\" (UID: \"a06e089e-fba5-403f-9fe5-34175c6865fa\") " Nov 03 02:01:46.376904 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:46.376583 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a06e089e-fba5-403f-9fe5-34175c6865fa-istio-data" pod "a06e089e-fba5-403f-9fe5-34175c6865fa" (UID: "a06e089e-fba5-403f-9fe5-34175c6865fa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:01:46.376904 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:46.376626 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a06e089e-fba5-403f-9fe5-34175c6865fa-workload-certs" pod "a06e089e-fba5-403f-9fe5-34175c6865fa" (UID: "a06e089e-fba5-403f-9fe5-34175c6865fa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:01:46.376904 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:46.376777 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a06e089e-fba5-403f-9fe5-34175c6865fa-credential-socket" pod "a06e089e-fba5-403f-9fe5-34175c6865fa" (UID: "a06e089e-fba5-403f-9fe5-34175c6865fa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:01:46.376904 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:46.376814 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a06e089e-fba5-403f-9fe5-34175c6865fa-workload-socket" pod "a06e089e-fba5-403f-9fe5-34175c6865fa" (UID: "a06e089e-fba5-403f-9fe5-34175c6865fa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:01:46.376904 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:46.376881 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a06e089e-fba5-403f-9fe5-34175c6865fa-istiod-ca-cert" pod "a06e089e-fba5-403f-9fe5-34175c6865fa" (UID: "a06e089e-fba5-403f-9fe5-34175c6865fa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:01:46.377444 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:46.377103 5437 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a06e089e-fba5-403f-9fe5-34175c6865fa-workload-certs\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:01:46.377444 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:46.377126 5437 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a06e089e-fba5-403f-9fe5-34175c6865fa-credential-socket\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:01:46.377444 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:46.377138 5437 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a06e089e-fba5-403f-9fe5-34175c6865fa-istiod-ca-cert\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:01:46.377444 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:46.377155 5437 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a06e089e-fba5-403f-9fe5-34175c6865fa-workload-socket\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:01:46.377444 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:46.377175 5437 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a06e089e-fba5-403f-9fe5-34175c6865fa-istio-data\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:01:46.378750 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:46.378719 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a06e089e-fba5-403f-9fe5-34175c6865fa-kube-api-access-t7l5x" pod "a06e089e-fba5-403f-9fe5-34175c6865fa" (UID: "a06e089e-fba5-403f-9fe5-34175c6865fa"). InnerVolumeSpecName "kube-api-access-t7l5x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:01:46.378939 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:46.378917 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a06e089e-fba5-403f-9fe5-34175c6865fa-istio-podinfo" pod "a06e089e-fba5-403f-9fe5-34175c6865fa" (UID: "a06e089e-fba5-403f-9fe5-34175c6865fa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:01:46.379191 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:46.379170 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a06e089e-fba5-403f-9fe5-34175c6865fa-istio-envoy" pod "a06e089e-fba5-403f-9fe5-34175c6865fa" (UID: "a06e089e-fba5-403f-9fe5-34175c6865fa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:01:46.379242 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:46.379208 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a06e089e-fba5-403f-9fe5-34175c6865fa-istio-token" pod "a06e089e-fba5-403f-9fe5-34175c6865fa" (UID: "a06e089e-fba5-403f-9fe5-34175c6865fa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:01:46.478094 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:46.478040 5437 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a06e089e-fba5-403f-9fe5-34175c6865fa-istio-envoy\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:01:46.478094 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:46.478091 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7l5x\" (UniqueName: \"kubernetes.io/projected/a06e089e-fba5-403f-9fe5-34175c6865fa-kube-api-access-t7l5x\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:01:46.478094 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:46.478104 5437 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a06e089e-fba5-403f-9fe5-34175c6865fa-istio-token\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:01:46.478094 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:46.478114 5437 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a06e089e-fba5-403f-9fe5-34175c6865fa-istio-podinfo\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:01:46.733457 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:46.733424 5437 generic.go:356] "Generic (PLEG): container finished" podID="a06e089e-fba5-403f-9fe5-34175c6865fa" containerID="fdded550c5957428c7f6d533d35877beeabf2bd33a185dc03a2ae3c1c1d821b2" exitCode=0 Nov 03 02:01:46.733676 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:46.733486 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-738acb-openshift-default-c7bc4fcf7-llk59" Nov 03 02:01:46.733676 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:46.733515 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-738acb-openshift-default-c7bc4fcf7-llk59" event={"ID":"a06e089e-fba5-403f-9fe5-34175c6865fa","Type":"ContainerDied","Data":"fdded550c5957428c7f6d533d35877beeabf2bd33a185dc03a2ae3c1c1d821b2"} Nov 03 02:01:46.733676 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:46.733585 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-738acb-openshift-default-c7bc4fcf7-llk59" event={"ID":"a06e089e-fba5-403f-9fe5-34175c6865fa","Type":"ContainerDied","Data":"c46faf9323563e066085344b2efdc5a74c122cb470433015f354672ad16e6038"} Nov 03 02:01:46.733676 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:46.733600 5437 scope.go:122] "RemoveContainer" containerID="fdded550c5957428c7f6d533d35877beeabf2bd33a185dc03a2ae3c1c1d821b2" Nov 03 02:01:46.746319 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:46.746295 5437 scope.go:122] "RemoveContainer" containerID="af249d8e97dad4103572480f15fd4ce304a5912a0d4aee49f3017d7960f2ecee" Nov 03 02:01:46.758175 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:46.758150 5437 scope.go:122] "RemoveContainer" containerID="fdded550c5957428c7f6d533d35877beeabf2bd33a185dc03a2ae3c1c1d821b2" Nov 03 02:01:46.758541 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:01:46.758519 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdded550c5957428c7f6d533d35877beeabf2bd33a185dc03a2ae3c1c1d821b2\": container with ID starting with fdded550c5957428c7f6d533d35877beeabf2bd33a185dc03a2ae3c1c1d821b2 not found: ID does not exist" containerID="fdded550c5957428c7f6d533d35877beeabf2bd33a185dc03a2ae3c1c1d821b2" Nov 03 02:01:46.758628 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:46.758554 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdded550c5957428c7f6d533d35877beeabf2bd33a185dc03a2ae3c1c1d821b2"} err="failed to get container status \"fdded550c5957428c7f6d533d35877beeabf2bd33a185dc03a2ae3c1c1d821b2\": rpc error: code = NotFound desc = could not find container \"fdded550c5957428c7f6d533d35877beeabf2bd33a185dc03a2ae3c1c1d821b2\": container with ID starting with fdded550c5957428c7f6d533d35877beeabf2bd33a185dc03a2ae3c1c1d821b2 not found: ID does not exist" Nov 03 02:01:46.758628 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:46.758587 5437 scope.go:122] "RemoveContainer" containerID="af249d8e97dad4103572480f15fd4ce304a5912a0d4aee49f3017d7960f2ecee" Nov 03 02:01:46.758928 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:01:46.758900 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af249d8e97dad4103572480f15fd4ce304a5912a0d4aee49f3017d7960f2ecee\": container with ID starting with af249d8e97dad4103572480f15fd4ce304a5912a0d4aee49f3017d7960f2ecee not found: ID does not exist" containerID="af249d8e97dad4103572480f15fd4ce304a5912a0d4aee49f3017d7960f2ecee" Nov 03 02:01:46.758969 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:46.758935 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af249d8e97dad4103572480f15fd4ce304a5912a0d4aee49f3017d7960f2ecee"} err="failed to get container status \"af249d8e97dad4103572480f15fd4ce304a5912a0d4aee49f3017d7960f2ecee\": rpc error: code = NotFound desc = could not find container \"af249d8e97dad4103572480f15fd4ce304a5912a0d4aee49f3017d7960f2ecee\": container with ID starting with af249d8e97dad4103572480f15fd4ce304a5912a0d4aee49f3017d7960f2ecee not found: ID does not exist" Nov 03 02:01:46.904322 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:46.904278 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-738acb-openshift-default-c7bc4fcf7-llk59"] Nov 03 02:01:46.910820 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:46.910785 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-738acb-openshift-default-c7bc4fcf7-llk59"] Nov 03 02:01:47.546299 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:47.546241 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:01:47.615757 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:47.615701 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-70235a-6f9d5df8dc-xgzc9" podUID="f1e4d46f-f257-45ea-8ffe-10e156db99ed" containerName="payload-processing" containerID="cri-o://ac8aaab0922763b48fe3601e143bf0be743ef0cf15d88835d7a117c52b717a63" gracePeriod=25 Nov 03 02:01:47.719497 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:47.719445 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-70235a-74f5459759-7hw6j" podUID="2e18e1b3-1ae6-4f7d-86ab-3821e7d5b72d" containerName="payload-pre-processing" containerID="cri-o://1f19395c3d50d99790a7db5acc59df89015c0dd73b6519052d4f0bfa6681547c" gracePeriod=25 Nov 03 02:01:47.744050 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:47.744009 5437 generic.go:356] "Generic (PLEG): container finished" podID="f1e4d46f-f257-45ea-8ffe-10e156db99ed" containerID="ac8aaab0922763b48fe3601e143bf0be743ef0cf15d88835d7a117c52b717a63" exitCode=0 Nov 03 02:01:47.744197 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:47.744080 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-70235a-6f9d5df8dc-xgzc9" event={"ID":"f1e4d46f-f257-45ea-8ffe-10e156db99ed","Type":"ContainerDied","Data":"ac8aaab0922763b48fe3601e143bf0be743ef0cf15d88835d7a117c52b717a63"} Nov 03 02:01:47.748936 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:47.748888 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:01:47.896228 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:47.896200 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-70235a-6f9d5df8dc-xgzc9" Nov 03 02:01:47.995022 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:47.994991 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-70235a-74f5459759-7hw6j" Nov 03 02:01:47.995235 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:47.995215 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1e4d46f-f257-45ea-8ffe-10e156db99ed-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1e4d46f-f257-45ea-8ffe-10e156db99ed-plugins-config-volume\") pod \"f1e4d46f-f257-45ea-8ffe-10e156db99ed\" (UID: \"f1e4d46f-f257-45ea-8ffe-10e156db99ed\") " Nov 03 02:01:47.995323 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:47.995309 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1e4d46f-f257-45ea-8ffe-10e156db99ed-kube-api-access-qzj4k\" (UniqueName: \"kubernetes.io/projected/f1e4d46f-f257-45ea-8ffe-10e156db99ed-kube-api-access-qzj4k\") pod \"f1e4d46f-f257-45ea-8ffe-10e156db99ed\" (UID: \"f1e4d46f-f257-45ea-8ffe-10e156db99ed\") " Nov 03 02:01:47.995600 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:47.995581 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1e4d46f-f257-45ea-8ffe-10e156db99ed-plugins-config-volume" pod "f1e4d46f-f257-45ea-8ffe-10e156db99ed" (UID: "f1e4d46f-f257-45ea-8ffe-10e156db99ed"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:01:47.997440 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:47.997421 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1e4d46f-f257-45ea-8ffe-10e156db99ed-kube-api-access-qzj4k" pod "f1e4d46f-f257-45ea-8ffe-10e156db99ed" (UID: "f1e4d46f-f257-45ea-8ffe-10e156db99ed"). InnerVolumeSpecName "kube-api-access-qzj4k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:01:48.089465 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:48.089428 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a06e089e-fba5-403f-9fe5-34175c6865fa" path="/var/lib/kubelet/pods/a06e089e-fba5-403f-9fe5-34175c6865fa/volumes" Nov 03 02:01:48.096974 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:48.096900 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e18e1b3-1ae6-4f7d-86ab-3821e7d5b72d-kube-api-access-sk86j\" (UniqueName: \"kubernetes.io/projected/2e18e1b3-1ae6-4f7d-86ab-3821e7d5b72d-kube-api-access-sk86j\") pod \"2e18e1b3-1ae6-4f7d-86ab-3821e7d5b72d\" (UID: \"2e18e1b3-1ae6-4f7d-86ab-3821e7d5b72d\") " Nov 03 02:01:48.097119 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:48.097058 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e18e1b3-1ae6-4f7d-86ab-3821e7d5b72d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e18e1b3-1ae6-4f7d-86ab-3821e7d5b72d-plugins-config-volume\") pod \"2e18e1b3-1ae6-4f7d-86ab-3821e7d5b72d\" (UID: \"2e18e1b3-1ae6-4f7d-86ab-3821e7d5b72d\") " Nov 03 02:01:48.097379 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:48.097352 5437 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1e4d46f-f257-45ea-8ffe-10e156db99ed-plugins-config-volume\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:01:48.097483 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:48.097380 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qzj4k\" (UniqueName: \"kubernetes.io/projected/f1e4d46f-f257-45ea-8ffe-10e156db99ed-kube-api-access-qzj4k\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:01:48.097483 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:48.097392 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e18e1b3-1ae6-4f7d-86ab-3821e7d5b72d-plugins-config-volume" pod "2e18e1b3-1ae6-4f7d-86ab-3821e7d5b72d" (UID: "2e18e1b3-1ae6-4f7d-86ab-3821e7d5b72d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:01:48.099438 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:48.099242 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e18e1b3-1ae6-4f7d-86ab-3821e7d5b72d-kube-api-access-sk86j" pod "2e18e1b3-1ae6-4f7d-86ab-3821e7d5b72d" (UID: "2e18e1b3-1ae6-4f7d-86ab-3821e7d5b72d"). InnerVolumeSpecName "kube-api-access-sk86j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:01:48.198424 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:48.198359 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sk86j\" (UniqueName: \"kubernetes.io/projected/2e18e1b3-1ae6-4f7d-86ab-3821e7d5b72d-kube-api-access-sk86j\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:01:48.198424 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:48.198413 5437 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e18e1b3-1ae6-4f7d-86ab-3821e7d5b72d-plugins-config-volume\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:01:48.751846 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:48.751816 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-70235a-6f9d5df8dc-xgzc9" Nov 03 02:01:48.751846 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:48.751837 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-70235a-6f9d5df8dc-xgzc9" event={"ID":"f1e4d46f-f257-45ea-8ffe-10e156db99ed","Type":"ContainerDied","Data":"13a58cf783e37194b692bccc6a9e46622eafaac4ff91c5cddb014f48bf8c8af8"} Nov 03 02:01:48.752408 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:48.751899 5437 scope.go:122] "RemoveContainer" containerID="ac8aaab0922763b48fe3601e143bf0be743ef0cf15d88835d7a117c52b717a63" Nov 03 02:01:48.753581 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:48.753555 5437 generic.go:356] "Generic (PLEG): container finished" podID="2e18e1b3-1ae6-4f7d-86ab-3821e7d5b72d" containerID="1f19395c3d50d99790a7db5acc59df89015c0dd73b6519052d4f0bfa6681547c" exitCode=0 Nov 03 02:01:48.753715 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:48.753603 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-70235a-74f5459759-7hw6j" Nov 03 02:01:48.753715 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:48.753646 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-70235a-74f5459759-7hw6j" event={"ID":"2e18e1b3-1ae6-4f7d-86ab-3821e7d5b72d","Type":"ContainerDied","Data":"1f19395c3d50d99790a7db5acc59df89015c0dd73b6519052d4f0bfa6681547c"} Nov 03 02:01:48.753821 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:48.753715 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-70235a-74f5459759-7hw6j" event={"ID":"2e18e1b3-1ae6-4f7d-86ab-3821e7d5b72d","Type":"ContainerDied","Data":"31709f0e4c57753bd3e0553b6eba971629cfb527f7596b066d57541ed7dc7930"} Nov 03 02:01:48.764294 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:48.764272 5437 scope.go:122] "RemoveContainer" containerID="1f19395c3d50d99790a7db5acc59df89015c0dd73b6519052d4f0bfa6681547c" Nov 03 02:01:48.773946 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:48.773921 5437 scope.go:122] "RemoveContainer" containerID="1f19395c3d50d99790a7db5acc59df89015c0dd73b6519052d4f0bfa6681547c" Nov 03 02:01:48.774281 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:01:48.774254 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f19395c3d50d99790a7db5acc59df89015c0dd73b6519052d4f0bfa6681547c\": container with ID starting with 1f19395c3d50d99790a7db5acc59df89015c0dd73b6519052d4f0bfa6681547c not found: ID does not exist" containerID="1f19395c3d50d99790a7db5acc59df89015c0dd73b6519052d4f0bfa6681547c" Nov 03 02:01:48.774476 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:48.774294 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f19395c3d50d99790a7db5acc59df89015c0dd73b6519052d4f0bfa6681547c"} err="failed to get container status \"1f19395c3d50d99790a7db5acc59df89015c0dd73b6519052d4f0bfa6681547c\": rpc error: code = NotFound desc = could not find container \"1f19395c3d50d99790a7db5acc59df89015c0dd73b6519052d4f0bfa6681547c\": container with ID starting with 1f19395c3d50d99790a7db5acc59df89015c0dd73b6519052d4f0bfa6681547c not found: ID does not exist" Nov 03 02:01:48.924122 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:48.924013 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-70235a-74f5459759-7hw6j"] Nov 03 02:01:48.973669 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:48.973625 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-70235a-74f5459759-7hw6j"] Nov 03 02:01:49.039691 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:49.039654 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-70235a-6f9d5df8dc-xgzc9"] Nov 03 02:01:49.043519 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:49.043476 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-70235a-6f9d5df8dc-xgzc9"] Nov 03 02:01:49.896492 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:49.896429 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-db583d-7cf47dd48c-lrzwz" podUID="768f8efe-c962-4d38-8743-a835df279bd7" containerName="payload-processing" containerID="cri-o://771b10b3356db42e6ec49621f7098d1294dc53e3e2cbe96222a3e0cd257f0dec" gracePeriod=25 Nov 03 02:01:50.090517 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:50.090482 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e18e1b3-1ae6-4f7d-86ab-3821e7d5b72d" path="/var/lib/kubelet/pods/2e18e1b3-1ae6-4f7d-86ab-3821e7d5b72d/volumes" Nov 03 02:01:50.090917 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:50.090903 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1e4d46f-f257-45ea-8ffe-10e156db99ed" path="/var/lib/kubelet/pods/f1e4d46f-f257-45ea-8ffe-10e156db99ed/volumes" Nov 03 02:01:50.174209 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:50.174175 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-db583d-7cf47dd48c-lrzwz" Nov 03 02:01:50.222360 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:50.222309 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/768f8efe-c962-4d38-8743-a835df279bd7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/768f8efe-c962-4d38-8743-a835df279bd7-plugins-config-volume\") pod \"768f8efe-c962-4d38-8743-a835df279bd7\" (UID: \"768f8efe-c962-4d38-8743-a835df279bd7\") " Nov 03 02:01:50.222360 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:50.222399 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/768f8efe-c962-4d38-8743-a835df279bd7-kube-api-access-wz6l9\" (UniqueName: \"kubernetes.io/projected/768f8efe-c962-4d38-8743-a835df279bd7-kube-api-access-wz6l9\") pod \"768f8efe-c962-4d38-8743-a835df279bd7\" (UID: \"768f8efe-c962-4d38-8743-a835df279bd7\") " Nov 03 02:01:50.222757 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:50.222716 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/768f8efe-c962-4d38-8743-a835df279bd7-plugins-config-volume" pod "768f8efe-c962-4d38-8743-a835df279bd7" (UID: "768f8efe-c962-4d38-8743-a835df279bd7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:01:50.224631 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:50.224602 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/768f8efe-c962-4d38-8743-a835df279bd7-kube-api-access-wz6l9" pod "768f8efe-c962-4d38-8743-a835df279bd7" (UID: "768f8efe-c962-4d38-8743-a835df279bd7"). InnerVolumeSpecName "kube-api-access-wz6l9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:01:50.324397 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:50.324336 5437 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/768f8efe-c962-4d38-8743-a835df279bd7-plugins-config-volume\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:01:50.324397 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:50.324390 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wz6l9\" (UniqueName: \"kubernetes.io/projected/768f8efe-c962-4d38-8743-a835df279bd7-kube-api-access-wz6l9\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:01:50.489171 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:50.489084 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:01:50.769159 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:50.769064 5437 generic.go:356] "Generic (PLEG): container finished" podID="768f8efe-c962-4d38-8743-a835df279bd7" containerID="771b10b3356db42e6ec49621f7098d1294dc53e3e2cbe96222a3e0cd257f0dec" exitCode=0 Nov 03 02:01:50.769159 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:50.769114 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-db583d-7cf47dd48c-lrzwz" Nov 03 02:01:50.769357 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:50.769152 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-db583d-7cf47dd48c-lrzwz" event={"ID":"768f8efe-c962-4d38-8743-a835df279bd7","Type":"ContainerDied","Data":"771b10b3356db42e6ec49621f7098d1294dc53e3e2cbe96222a3e0cd257f0dec"} Nov 03 02:01:50.769357 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:50.769214 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-db583d-7cf47dd48c-lrzwz" event={"ID":"768f8efe-c962-4d38-8743-a835df279bd7","Type":"ContainerDied","Data":"eba1dac4798fd420fa66db7e58a3a6fb90c8f5546eff1c61125537fe86698401"} Nov 03 02:01:50.769357 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:50.769243 5437 scope.go:122] "RemoveContainer" containerID="771b10b3356db42e6ec49621f7098d1294dc53e3e2cbe96222a3e0cd257f0dec" Nov 03 02:01:50.779563 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:50.779535 5437 scope.go:122] "RemoveContainer" containerID="771b10b3356db42e6ec49621f7098d1294dc53e3e2cbe96222a3e0cd257f0dec" Nov 03 02:01:50.780003 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:01:50.779975 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"771b10b3356db42e6ec49621f7098d1294dc53e3e2cbe96222a3e0cd257f0dec\": container with ID starting with 771b10b3356db42e6ec49621f7098d1294dc53e3e2cbe96222a3e0cd257f0dec not found: ID does not exist" containerID="771b10b3356db42e6ec49621f7098d1294dc53e3e2cbe96222a3e0cd257f0dec" Nov 03 02:01:50.780122 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:50.780016 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"771b10b3356db42e6ec49621f7098d1294dc53e3e2cbe96222a3e0cd257f0dec"} err="failed to get container status \"771b10b3356db42e6ec49621f7098d1294dc53e3e2cbe96222a3e0cd257f0dec\": rpc error: code = NotFound desc = could not find container \"771b10b3356db42e6ec49621f7098d1294dc53e3e2cbe96222a3e0cd257f0dec\": container with ID starting with 771b10b3356db42e6ec49621f7098d1294dc53e3e2cbe96222a3e0cd257f0dec not found: ID does not exist" Nov 03 02:01:50.938993 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:50.938957 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-db583d-7cf47dd48c-lrzwz"] Nov 03 02:01:50.942306 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:50.942272 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-db583d-7cf47dd48c-lrzwz"] Nov 03 02:01:51.560182 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:51.560144 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:01:52.088501 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:52.088466 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="768f8efe-c962-4d38-8743-a835df279bd7" path="/var/lib/kubelet/pods/768f8efe-c962-4d38-8743-a835df279bd7/volumes" Nov 03 02:01:52.618400 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:52.618346 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:01:55.127943 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:55.127899 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:01:57.431058 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:01:57.431019 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:02:01.661175 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:01.661101 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:02:02.598460 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:02.598420 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-70235a-545895b546-88g4c"] Nov 03 02:02:02.598689 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:02.598667 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-70235a-545895b546-88g4c" podUID="51aabea8-7bcc-49d2-bf06-6b8ba825fd33" containerName="maas-api" containerID="cri-o://75df0b4c7c96f5fb902de6e6680c5c34496f7e1ad0a4a9a49ac123b0ed7c4a69" gracePeriod=30 Nov 03 02:02:02.754251 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:02.754206 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:02:02.840488 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:02.840448 5437 generic.go:356] "Generic (PLEG): container finished" podID="51aabea8-7bcc-49d2-bf06-6b8ba825fd33" containerID="75df0b4c7c96f5fb902de6e6680c5c34496f7e1ad0a4a9a49ac123b0ed7c4a69" exitCode=0 Nov 03 02:02:02.840674 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:02.840530 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-70235a-545895b546-88g4c" event={"ID":"51aabea8-7bcc-49d2-bf06-6b8ba825fd33","Type":"ContainerDied","Data":"75df0b4c7c96f5fb902de6e6680c5c34496f7e1ad0a4a9a49ac123b0ed7c4a69"} Nov 03 02:02:02.868232 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:02.868196 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-70235a-545895b546-88g4c" Nov 03 02:02:02.972769 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:02.972736 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51aabea8-7bcc-49d2-bf06-6b8ba825fd33-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51aabea8-7bcc-49d2-bf06-6b8ba825fd33-openshift-service-ca\") pod \"51aabea8-7bcc-49d2-bf06-6b8ba825fd33\" (UID: \"51aabea8-7bcc-49d2-bf06-6b8ba825fd33\") " Nov 03 02:02:02.972949 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:02.972840 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51aabea8-7bcc-49d2-bf06-6b8ba825fd33-kube-api-access-4hr99\" (UniqueName: \"kubernetes.io/projected/51aabea8-7bcc-49d2-bf06-6b8ba825fd33-kube-api-access-4hr99\") pod \"51aabea8-7bcc-49d2-bf06-6b8ba825fd33\" (UID: \"51aabea8-7bcc-49d2-bf06-6b8ba825fd33\") " Nov 03 02:02:02.972949 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:02.972892 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51aabea8-7bcc-49d2-bf06-6b8ba825fd33-metrics-certs\" (UniqueName: \"kubernetes.io/secret/51aabea8-7bcc-49d2-bf06-6b8ba825fd33-metrics-certs\") pod \"51aabea8-7bcc-49d2-bf06-6b8ba825fd33\" (UID: \"51aabea8-7bcc-49d2-bf06-6b8ba825fd33\") " Nov 03 02:02:02.973094 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:02.973003 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51aabea8-7bcc-49d2-bf06-6b8ba825fd33-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51aabea8-7bcc-49d2-bf06-6b8ba825fd33-maas-api-tls\") pod \"51aabea8-7bcc-49d2-bf06-6b8ba825fd33\" (UID: \"51aabea8-7bcc-49d2-bf06-6b8ba825fd33\") " Nov 03 02:02:02.973329 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:02.973302 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51aabea8-7bcc-49d2-bf06-6b8ba825fd33-openshift-service-ca" pod "51aabea8-7bcc-49d2-bf06-6b8ba825fd33" (UID: "51aabea8-7bcc-49d2-bf06-6b8ba825fd33"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:02:02.973564 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:02.973537 5437 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51aabea8-7bcc-49d2-bf06-6b8ba825fd33-openshift-service-ca\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:02:02.975548 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:02.975516 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51aabea8-7bcc-49d2-bf06-6b8ba825fd33-maas-api-tls" pod "51aabea8-7bcc-49d2-bf06-6b8ba825fd33" (UID: "51aabea8-7bcc-49d2-bf06-6b8ba825fd33"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:02:02.975659 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:02.975569 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51aabea8-7bcc-49d2-bf06-6b8ba825fd33-kube-api-access-4hr99" pod "51aabea8-7bcc-49d2-bf06-6b8ba825fd33" (UID: "51aabea8-7bcc-49d2-bf06-6b8ba825fd33"). InnerVolumeSpecName "kube-api-access-4hr99". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:02:02.976474 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:02.976450 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51aabea8-7bcc-49d2-bf06-6b8ba825fd33-metrics-certs" pod "51aabea8-7bcc-49d2-bf06-6b8ba825fd33" (UID: "51aabea8-7bcc-49d2-bf06-6b8ba825fd33"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:02:03.074711 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:03.074669 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hr99\" (UniqueName: \"kubernetes.io/projected/51aabea8-7bcc-49d2-bf06-6b8ba825fd33-kube-api-access-4hr99\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:02:03.074711 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:03.074706 5437 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51aabea8-7bcc-49d2-bf06-6b8ba825fd33-metrics-certs\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:02:03.074940 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:03.074729 5437 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51aabea8-7bcc-49d2-bf06-6b8ba825fd33-maas-api-tls\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:02:03.845305 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:03.845246 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:02:03.848566 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:03.848532 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-70235a-545895b546-88g4c" event={"ID":"51aabea8-7bcc-49d2-bf06-6b8ba825fd33","Type":"ContainerDied","Data":"9dddce151ed19d54a4d28299f6ff9474f0f54f2c7568f9db5eb2b7a05c7effd4"} Nov 03 02:02:03.848726 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:03.848591 5437 scope.go:122] "RemoveContainer" containerID="75df0b4c7c96f5fb902de6e6680c5c34496f7e1ad0a4a9a49ac123b0ed7c4a69" Nov 03 02:02:03.848726 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:03.848542 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-70235a-545895b546-88g4c" Nov 03 02:02:04.018808 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:04.018768 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-70235a-545895b546-88g4c"] Nov 03 02:02:04.070347 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:04.070308 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-70235a-545895b546-88g4c"] Nov 03 02:02:04.088529 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:04.088487 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51aabea8-7bcc-49d2-bf06-6b8ba825fd33" path="/var/lib/kubelet/pods/51aabea8-7bcc-49d2-bf06-6b8ba825fd33/volumes" Nov 03 02:02:04.337079 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:04.337038 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:02:04.539201 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:04.539033 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:02:05.577970 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:05.577921 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:02:05.929096 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:05.929002 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:02:06.136655 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:06.136617 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:02:07.484986 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:07.484941 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:02:07.734236 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:07.734190 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:02:07.937468 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:07.937432 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:02:08.135420 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:08.135354 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:02:08.932389 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:08.932324 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:02:09.958866 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:09.958822 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:02:10.211209 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:10.211115 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:02:10.701048 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:10.701013 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:02:11.223714 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:11.223678 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:02:11.427746 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:11.427708 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:02:11.666284 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:11.666236 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:02:11.996706 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:11.996604 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:02:12.824593 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:12.824552 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:02:13.345760 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:13.345720 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:02:14.035764 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:02:14.035724 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 02:02:14.171415 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:14.171351 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:02:15.034409 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:15.034349 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:02:15.246456 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:15.246412 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:02:17.230914 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:17.230828 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:02:20.225122 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:20.225090 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-e0907d-74b7bd8f9f-pdk27"] Nov 03 02:02:20.635278 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:20.635236 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:02:20.874633 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:20.874586 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:02:21.242144 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:21.242104 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:02:21.524337 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:21.524249 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:02:22.383518 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:22.383467 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:02:22.467268 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:22.467232 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-b2x7h"] Nov 03 02:02:22.467823 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:22.467801 5437 cpu_manager.go:414] "containerMap: removing container" podUID="56f66e18-88ac-4524-aee2-ebec8cce6dd8" containerName="cleanup" Nov 03 02:02:22.467823 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:22.467822 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56f66e18-88ac-4524-aee2-ebec8cce6dd8" containerName="cleanup" Nov 03 02:02:22.467973 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:22.467837 5437 cpu_manager.go:414] "containerMap: removing container" podUID="a06e089e-fba5-403f-9fe5-34175c6865fa" containerName="istio-proxy" Nov 03 02:02:22.467973 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:22.467843 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a06e089e-fba5-403f-9fe5-34175c6865fa" containerName="istio-proxy" Nov 03 02:02:22.467973 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:22.467852 5437 cpu_manager.go:414] "containerMap: removing container" podUID="768f8efe-c962-4d38-8743-a835df279bd7" containerName="payload-processing" Nov 03 02:02:22.467973 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:22.467857 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="768f8efe-c962-4d38-8743-a835df279bd7" containerName="payload-processing" Nov 03 02:02:22.467973 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:22.467868 5437 cpu_manager.go:414] "containerMap: removing container" podUID="f1e4d46f-f257-45ea-8ffe-10e156db99ed" containerName="payload-processing" Nov 03 02:02:22.467973 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:22.467873 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1e4d46f-f257-45ea-8ffe-10e156db99ed" containerName="payload-processing" Nov 03 02:02:22.467973 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:22.467882 5437 cpu_manager.go:414] "containerMap: removing container" podUID="306b59aa-822c-4e6c-9434-bc06393c2802" containerName="maas-api" Nov 03 02:02:22.467973 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:22.467887 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="306b59aa-822c-4e6c-9434-bc06393c2802" containerName="maas-api" Nov 03 02:02:22.467973 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:22.467895 5437 cpu_manager.go:414] "containerMap: removing container" podUID="9b955a41-8e08-46b0-87d0-64ab0cc1e381" containerName="istio-proxy" Nov 03 02:02:22.467973 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:22.467899 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b955a41-8e08-46b0-87d0-64ab0cc1e381" containerName="istio-proxy" Nov 03 02:02:22.467973 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:22.467908 5437 cpu_manager.go:414] "containerMap: removing container" podUID="56f66e18-88ac-4524-aee2-ebec8cce6dd8" containerName="cleanup" Nov 03 02:02:22.467973 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:22.467914 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56f66e18-88ac-4524-aee2-ebec8cce6dd8" containerName="cleanup" Nov 03 02:02:22.467973 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:22.467921 5437 cpu_manager.go:414] "containerMap: removing container" podUID="cb08db38-b50e-4488-ada1-ffce3cf76fd1" containerName="cleanup" Nov 03 02:02:22.467973 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:22.467926 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb08db38-b50e-4488-ada1-ffce3cf76fd1" containerName="cleanup" Nov 03 02:02:22.467973 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:22.467934 5437 cpu_manager.go:414] "containerMap: removing container" podUID="51aabea8-7bcc-49d2-bf06-6b8ba825fd33" containerName="maas-api" Nov 03 02:02:22.467973 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:22.467938 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51aabea8-7bcc-49d2-bf06-6b8ba825fd33" containerName="maas-api" Nov 03 02:02:22.467973 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:22.467945 5437 cpu_manager.go:414] "containerMap: removing container" podUID="cb08db38-b50e-4488-ada1-ffce3cf76fd1" containerName="cleanup" Nov 03 02:02:22.467973 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:22.467950 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb08db38-b50e-4488-ada1-ffce3cf76fd1" containerName="cleanup" Nov 03 02:02:22.467973 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:22.467959 5437 cpu_manager.go:414] "containerMap: removing container" podUID="a255ffd8-6415-41b7-af49-0e6522ff1133" containerName="payload-pre-processing" Nov 03 02:02:22.467973 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:22.467964 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a255ffd8-6415-41b7-af49-0e6522ff1133" containerName="payload-pre-processing" Nov 03 02:02:22.467973 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:22.467976 5437 cpu_manager.go:414] "containerMap: removing container" podUID="2e18e1b3-1ae6-4f7d-86ab-3821e7d5b72d" containerName="payload-pre-processing" Nov 03 02:02:22.467973 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:22.467982 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e18e1b3-1ae6-4f7d-86ab-3821e7d5b72d" containerName="payload-pre-processing" Nov 03 02:02:22.469095 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:22.467992 5437 cpu_manager.go:414] "containerMap: removing container" podUID="a06e089e-fba5-403f-9fe5-34175c6865fa" containerName="istio-proxy" Nov 03 02:02:22.469095 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:22.467998 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a06e089e-fba5-403f-9fe5-34175c6865fa" containerName="istio-proxy" Nov 03 02:02:22.469095 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:22.468004 5437 cpu_manager.go:414] "containerMap: removing container" podUID="86c42961-145d-4e9d-b66e-a0bc06cc6065" containerName="cleanup" Nov 03 02:02:22.469095 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:22.468009 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86c42961-145d-4e9d-b66e-a0bc06cc6065" containerName="cleanup" Nov 03 02:02:22.469095 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:22.468015 5437 cpu_manager.go:414] "containerMap: removing container" podUID="86c42961-145d-4e9d-b66e-a0bc06cc6065" containerName="cleanup" Nov 03 02:02:22.469095 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:22.468019 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86c42961-145d-4e9d-b66e-a0bc06cc6065" containerName="cleanup" Nov 03 02:02:22.469095 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:22.468027 5437 cpu_manager.go:414] "containerMap: removing container" podUID="775139d4-2b62-484e-8d22-ae25e9e2d4d5" containerName="payload-processing" Nov 03 02:02:22.469095 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:22.468031 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="775139d4-2b62-484e-8d22-ae25e9e2d4d5" containerName="payload-processing" Nov 03 02:02:22.469095 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:22.468041 5437 cpu_manager.go:414] "containerMap: removing container" podUID="86c42961-145d-4e9d-b66e-a0bc06cc6065" containerName="cleanup" Nov 03 02:02:22.469095 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:22.468045 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86c42961-145d-4e9d-b66e-a0bc06cc6065" containerName="cleanup" Nov 03 02:02:22.469095 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:22.468124 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e18e1b3-1ae6-4f7d-86ab-3821e7d5b72d" containerName="payload-pre-processing" Nov 03 02:02:22.469095 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:22.468133 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="a06e089e-fba5-403f-9fe5-34175c6865fa" containerName="istio-proxy" Nov 03 02:02:22.469095 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:22.468140 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="86c42961-145d-4e9d-b66e-a0bc06cc6065" containerName="cleanup" Nov 03 02:02:22.469095 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:22.468144 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="86c42961-145d-4e9d-b66e-a0bc06cc6065" containerName="cleanup" Nov 03 02:02:22.469095 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:22.468151 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="775139d4-2b62-484e-8d22-ae25e9e2d4d5" containerName="payload-processing" Nov 03 02:02:22.469095 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:22.468155 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="56f66e18-88ac-4524-aee2-ebec8cce6dd8" containerName="cleanup" Nov 03 02:02:22.469095 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:22.468160 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb08db38-b50e-4488-ada1-ffce3cf76fd1" containerName="cleanup" Nov 03 02:02:22.469095 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:22.468165 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="86c42961-145d-4e9d-b66e-a0bc06cc6065" containerName="cleanup" Nov 03 02:02:22.469095 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:22.468173 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="56f66e18-88ac-4524-aee2-ebec8cce6dd8" containerName="cleanup" Nov 03 02:02:22.469095 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:22.468178 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="768f8efe-c962-4d38-8743-a835df279bd7" containerName="payload-processing" Nov 03 02:02:22.469095 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:22.468182 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1e4d46f-f257-45ea-8ffe-10e156db99ed" containerName="payload-processing" Nov 03 02:02:22.469095 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:22.468189 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="306b59aa-822c-4e6c-9434-bc06393c2802" containerName="maas-api" Nov 03 02:02:22.469095 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:22.468195 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b955a41-8e08-46b0-87d0-64ab0cc1e381" containerName="istio-proxy" Nov 03 02:02:22.469095 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:22.468200 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb08db38-b50e-4488-ada1-ffce3cf76fd1" containerName="cleanup" Nov 03 02:02:22.469095 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:22.468204 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="51aabea8-7bcc-49d2-bf06-6b8ba825fd33" containerName="maas-api" Nov 03 02:02:22.469095 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:22.468209 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb08db38-b50e-4488-ada1-ffce3cf76fd1" containerName="cleanup" Nov 03 02:02:22.469095 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:22.468216 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="a255ffd8-6415-41b7-af49-0e6522ff1133" containerName="payload-pre-processing" Nov 03 02:02:22.473437 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:22.473402 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-b2x7h" Nov 03 02:02:22.527386 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:22.527326 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-9da431-openshift-default-dockercfg-bq94q\"" Nov 03 02:02:22.581442 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:22.581406 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-b2x7h"] Nov 03 02:02:22.608508 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:22.608462 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d8023ca-6b91-43aa-b753-9ff28068ca28-credential-socket\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-b2x7h\" (UID: \"8d8023ca-6b91-43aa-b753-9ff28068ca28\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-b2x7h" Nov 03 02:02:22.608711 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:22.608570 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d8023ca-6b91-43aa-b753-9ff28068ca28-istio-token\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-b2x7h\" (UID: \"8d8023ca-6b91-43aa-b753-9ff28068ca28\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-b2x7h" Nov 03 02:02:22.608711 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:22.608644 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d8023ca-6b91-43aa-b753-9ff28068ca28-workload-socket\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-b2x7h\" (UID: \"8d8023ca-6b91-43aa-b753-9ff28068ca28\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-b2x7h" Nov 03 02:02:22.608711 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:22.608677 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d8023ca-6b91-43aa-b753-9ff28068ca28-istiod-ca-cert\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-b2x7h\" (UID: \"8d8023ca-6b91-43aa-b753-9ff28068ca28\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-b2x7h" Nov 03 02:02:22.608818 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:22.608803 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d8023ca-6b91-43aa-b753-9ff28068ca28-istio-podinfo\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-b2x7h\" (UID: \"8d8023ca-6b91-43aa-b753-9ff28068ca28\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-b2x7h" Nov 03 02:02:22.608859 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:22.608833 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d8023ca-6b91-43aa-b753-9ff28068ca28-istio-data\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-b2x7h\" (UID: \"8d8023ca-6b91-43aa-b753-9ff28068ca28\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-b2x7h" Nov 03 02:02:22.608893 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:22.608859 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d8023ca-6b91-43aa-b753-9ff28068ca28-istio-envoy\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-b2x7h\" (UID: \"8d8023ca-6b91-43aa-b753-9ff28068ca28\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-b2x7h" Nov 03 02:02:22.608927 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:22.608914 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v75tj\" (UniqueName: \"kubernetes.io/projected/8d8023ca-6b91-43aa-b753-9ff28068ca28-kube-api-access-v75tj\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-b2x7h\" (UID: \"8d8023ca-6b91-43aa-b753-9ff28068ca28\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-b2x7h" Nov 03 02:02:22.608965 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:22.608955 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d8023ca-6b91-43aa-b753-9ff28068ca28-workload-certs\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-b2x7h\" (UID: \"8d8023ca-6b91-43aa-b753-9ff28068ca28\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-b2x7h" Nov 03 02:02:22.709729 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:22.709642 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d8023ca-6b91-43aa-b753-9ff28068ca28-istio-podinfo\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-b2x7h\" (UID: \"8d8023ca-6b91-43aa-b753-9ff28068ca28\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-b2x7h" Nov 03 02:02:22.709729 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:22.709683 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d8023ca-6b91-43aa-b753-9ff28068ca28-istio-data\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-b2x7h\" (UID: \"8d8023ca-6b91-43aa-b753-9ff28068ca28\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-b2x7h" Nov 03 02:02:22.709729 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:22.709704 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d8023ca-6b91-43aa-b753-9ff28068ca28-istio-envoy\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-b2x7h\" (UID: \"8d8023ca-6b91-43aa-b753-9ff28068ca28\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-b2x7h" Nov 03 02:02:22.709729 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:22.709724 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v75tj\" (UniqueName: \"kubernetes.io/projected/8d8023ca-6b91-43aa-b753-9ff28068ca28-kube-api-access-v75tj\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-b2x7h\" (UID: \"8d8023ca-6b91-43aa-b753-9ff28068ca28\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-b2x7h" Nov 03 02:02:22.710083 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:22.709755 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d8023ca-6b91-43aa-b753-9ff28068ca28-workload-certs\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-b2x7h\" (UID: \"8d8023ca-6b91-43aa-b753-9ff28068ca28\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-b2x7h" Nov 03 02:02:22.710083 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:22.709801 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d8023ca-6b91-43aa-b753-9ff28068ca28-credential-socket\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-b2x7h\" (UID: \"8d8023ca-6b91-43aa-b753-9ff28068ca28\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-b2x7h" Nov 03 02:02:22.710083 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:22.709836 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d8023ca-6b91-43aa-b753-9ff28068ca28-istio-token\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-b2x7h\" (UID: \"8d8023ca-6b91-43aa-b753-9ff28068ca28\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-b2x7h" Nov 03 02:02:22.710083 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:22.709872 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d8023ca-6b91-43aa-b753-9ff28068ca28-workload-socket\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-b2x7h\" (UID: \"8d8023ca-6b91-43aa-b753-9ff28068ca28\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-b2x7h" Nov 03 02:02:22.710083 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:22.709910 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d8023ca-6b91-43aa-b753-9ff28068ca28-istiod-ca-cert\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-b2x7h\" (UID: \"8d8023ca-6b91-43aa-b753-9ff28068ca28\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-b2x7h" Nov 03 02:02:22.710354 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:22.710234 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d8023ca-6b91-43aa-b753-9ff28068ca28-credential-socket\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-b2x7h\" (UID: \"8d8023ca-6b91-43aa-b753-9ff28068ca28\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-b2x7h" Nov 03 02:02:22.710354 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:22.710284 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d8023ca-6b91-43aa-b753-9ff28068ca28-workload-certs\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-b2x7h\" (UID: \"8d8023ca-6b91-43aa-b753-9ff28068ca28\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-b2x7h" Nov 03 02:02:22.710354 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:22.710316 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d8023ca-6b91-43aa-b753-9ff28068ca28-workload-socket\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-b2x7h\" (UID: \"8d8023ca-6b91-43aa-b753-9ff28068ca28\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-b2x7h" Nov 03 02:02:22.710546 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:22.710442 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d8023ca-6b91-43aa-b753-9ff28068ca28-istio-data\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-b2x7h\" (UID: \"8d8023ca-6b91-43aa-b753-9ff28068ca28\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-b2x7h" Nov 03 02:02:22.710653 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:22.710633 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d8023ca-6b91-43aa-b753-9ff28068ca28-istiod-ca-cert\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-b2x7h\" (UID: \"8d8023ca-6b91-43aa-b753-9ff28068ca28\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-b2x7h" Nov 03 02:02:22.712177 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:22.712157 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d8023ca-6b91-43aa-b753-9ff28068ca28-istio-envoy\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-b2x7h\" (UID: \"8d8023ca-6b91-43aa-b753-9ff28068ca28\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-b2x7h" Nov 03 02:02:22.712554 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:22.712534 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d8023ca-6b91-43aa-b753-9ff28068ca28-istio-podinfo\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-b2x7h\" (UID: \"8d8023ca-6b91-43aa-b753-9ff28068ca28\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-b2x7h" Nov 03 02:02:22.743871 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:22.743834 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:02:22.772793 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:22.772750 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d8023ca-6b91-43aa-b753-9ff28068ca28-istio-token\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-b2x7h\" (UID: \"8d8023ca-6b91-43aa-b753-9ff28068ca28\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-b2x7h" Nov 03 02:02:22.772954 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:22.772875 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v75tj\" (UniqueName: \"kubernetes.io/projected/8d8023ca-6b91-43aa-b753-9ff28068ca28-kube-api-access-v75tj\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-b2x7h\" (UID: \"8d8023ca-6b91-43aa-b753-9ff28068ca28\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-b2x7h" Nov 03 02:02:22.793727 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:22.793685 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-b2x7h" Nov 03 02:02:23.220473 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:23.220439 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-b2x7h"] Nov 03 02:02:23.222679 ip-10-0-136-16 kubenswrapper[5437]: W1103 02:02:23.222651 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d8023ca_6b91_43aa_b753_9ff28068ca28.slice/crio-04b8acd0872a23d1960d442b86a095706433933ffeb6e30af21173fa2d8137f0 WatchSource:0}: Error finding container 04b8acd0872a23d1960d442b86a095706433933ffeb6e30af21173fa2d8137f0: Status 404 returned error can't find the container with id 04b8acd0872a23d1960d442b86a095706433933ffeb6e30af21173fa2d8137f0 Nov 03 02:02:23.225995 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:23.225944 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 02:02:23.226101 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:23.226049 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 02:02:23.226101 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:23.226092 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 02:02:23.328280 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:23.328229 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:02:23.972651 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:23.972615 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-b2x7h" event={"ID":"8d8023ca-6b91-43aa-b753-9ff28068ca28","Type":"ContainerStarted","Data":"b5d7e6a5d8209bdeeea0e078af7684e0bc0a1e662de5bb7ed153d87356547bea"} Nov 03 02:02:23.973112 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:23.972676 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-b2x7h" event={"ID":"8d8023ca-6b91-43aa-b753-9ff28068ca28","Type":"ContainerStarted","Data":"04b8acd0872a23d1960d442b86a095706433933ffeb6e30af21173fa2d8137f0"} Nov 03 02:02:24.096504 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:24.096436 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-b2x7h" podStartSLOduration=2.096417419 podStartE2EDuration="2.096417419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:02:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:02:24.095820687 +0000 UTC m=+2626.125302992" watchObservedRunningTime="2026-11-03 02:02:24.096417419 +0000 UTC m=+2626.125899715" Nov 03 02:02:24.794397 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:24.794343 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-b2x7h" Nov 03 02:02:24.796033 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:24.796003 5437 patch_prober.go:29] interesting pod/e2e-worker-w2-9da431-openshift-default-65c799fd77-b2x7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Nov 03 02:02:24.796172 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:24.796056 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-b2x7h" podUID="8d8023ca-6b91-43aa-b753-9ff28068ca28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Nov 03 02:02:25.208492 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:25.208457 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:02:25.242573 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:25.242441 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-e0907d-74b7bd8f9f-pdk27" podUID="653ed959-fdf9-4ad7-90be-0f58348c80a3" containerName="payload-processing" containerID="cri-o://97e434e0da02e2b9bfc8972e7dc4ab769765378c860171b3d5f587b93eaafba3" gracePeriod=25 Nov 03 02:02:25.556547 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:25.556515 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-e0907d-74b7bd8f9f-pdk27" Nov 03 02:02:25.657027 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:25.656989 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/653ed959-fdf9-4ad7-90be-0f58348c80a3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/653ed959-fdf9-4ad7-90be-0f58348c80a3-plugins-config-volume\") pod \"653ed959-fdf9-4ad7-90be-0f58348c80a3\" (UID: \"653ed959-fdf9-4ad7-90be-0f58348c80a3\") " Nov 03 02:02:25.657212 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:25.657082 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/653ed959-fdf9-4ad7-90be-0f58348c80a3-kube-api-access-55jbp\" (UniqueName: \"kubernetes.io/projected/653ed959-fdf9-4ad7-90be-0f58348c80a3-kube-api-access-55jbp\") pod \"653ed959-fdf9-4ad7-90be-0f58348c80a3\" (UID: \"653ed959-fdf9-4ad7-90be-0f58348c80a3\") " Nov 03 02:02:25.657424 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:25.657400 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/653ed959-fdf9-4ad7-90be-0f58348c80a3-plugins-config-volume" pod "653ed959-fdf9-4ad7-90be-0f58348c80a3" (UID: "653ed959-fdf9-4ad7-90be-0f58348c80a3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:02:25.657596 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:25.657582 5437 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/653ed959-fdf9-4ad7-90be-0f58348c80a3-plugins-config-volume\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:02:25.659296 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:25.659261 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/653ed959-fdf9-4ad7-90be-0f58348c80a3-kube-api-access-55jbp" pod "653ed959-fdf9-4ad7-90be-0f58348c80a3" (UID: "653ed959-fdf9-4ad7-90be-0f58348c80a3"). InnerVolumeSpecName "kube-api-access-55jbp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:02:25.759614 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:25.758818 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55jbp\" (UniqueName: \"kubernetes.io/projected/653ed959-fdf9-4ad7-90be-0f58348c80a3-kube-api-access-55jbp\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:02:25.794307 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:25.794201 5437 patch_prober.go:29] interesting pod/e2e-worker-w2-9da431-openshift-default-65c799fd77-b2x7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Nov 03 02:02:25.794307 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:25.794260 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-b2x7h" podUID="8d8023ca-6b91-43aa-b753-9ff28068ca28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Nov 03 02:02:25.888209 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:25.888174 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:02:25.985707 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:25.985677 5437 generic.go:356] "Generic (PLEG): container finished" podID="653ed959-fdf9-4ad7-90be-0f58348c80a3" containerID="97e434e0da02e2b9bfc8972e7dc4ab769765378c860171b3d5f587b93eaafba3" exitCode=0 Nov 03 02:02:25.985989 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:25.985725 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-e0907d-74b7bd8f9f-pdk27" Nov 03 02:02:25.986070 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:25.985745 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-e0907d-74b7bd8f9f-pdk27" event={"ID":"653ed959-fdf9-4ad7-90be-0f58348c80a3","Type":"ContainerDied","Data":"97e434e0da02e2b9bfc8972e7dc4ab769765378c860171b3d5f587b93eaafba3"} Nov 03 02:02:25.986136 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:25.986116 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-e0907d-74b7bd8f9f-pdk27" event={"ID":"653ed959-fdf9-4ad7-90be-0f58348c80a3","Type":"ContainerDied","Data":"2e932723d19d88adfbe0d17f3b0eaf8a40691500413d86e6d8a04fce2588a1dc"} Nov 03 02:02:25.986176 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:25.986150 5437 scope.go:122] "RemoveContainer" containerID="97e434e0da02e2b9bfc8972e7dc4ab769765378c860171b3d5f587b93eaafba3" Nov 03 02:02:26.001179 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:26.001136 5437 scope.go:122] "RemoveContainer" containerID="97e434e0da02e2b9bfc8972e7dc4ab769765378c860171b3d5f587b93eaafba3" Nov 03 02:02:26.001930 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:02:26.001904 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97e434e0da02e2b9bfc8972e7dc4ab769765378c860171b3d5f587b93eaafba3\": container with ID starting with 97e434e0da02e2b9bfc8972e7dc4ab769765378c860171b3d5f587b93eaafba3 not found: ID does not exist" containerID="97e434e0da02e2b9bfc8972e7dc4ab769765378c860171b3d5f587b93eaafba3" Nov 03 02:02:26.002031 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:26.001937 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97e434e0da02e2b9bfc8972e7dc4ab769765378c860171b3d5f587b93eaafba3"} err="failed to get container status \"97e434e0da02e2b9bfc8972e7dc4ab769765378c860171b3d5f587b93eaafba3\": rpc error: code = NotFound desc = could not find container \"97e434e0da02e2b9bfc8972e7dc4ab769765378c860171b3d5f587b93eaafba3\": container with ID starting with 97e434e0da02e2b9bfc8972e7dc4ab769765378c860171b3d5f587b93eaafba3 not found: ID does not exist" Nov 03 02:02:26.162265 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:26.162223 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-e0907d-74b7bd8f9f-pdk27"] Nov 03 02:02:26.168354 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:26.168322 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-e0907d-74b7bd8f9f-pdk27"] Nov 03 02:02:26.306733 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:26.306693 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:02:26.795049 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:26.795010 5437 patch_prober.go:29] interesting pod/e2e-worker-w2-9da431-openshift-default-65c799fd77-b2x7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Nov 03 02:02:26.795211 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:26.795066 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-b2x7h" podUID="8d8023ca-6b91-43aa-b753-9ff28068ca28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Nov 03 02:02:26.894150 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:26.894118 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:02:27.794662 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:27.794633 5437 patch_prober.go:29] interesting pod/e2e-worker-w2-9da431-openshift-default-65c799fd77-b2x7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Nov 03 02:02:27.795145 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:27.795122 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-b2x7h" podUID="8d8023ca-6b91-43aa-b753-9ff28068ca28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Nov 03 02:02:27.849910 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:27.849872 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:02:28.089076 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:28.089043 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="653ed959-fdf9-4ad7-90be-0f58348c80a3" path="/var/lib/kubelet/pods/653ed959-fdf9-4ad7-90be-0f58348c80a3/volumes" Nov 03 02:02:28.794446 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:28.794352 5437 patch_prober.go:29] interesting pod/e2e-worker-w2-9da431-openshift-default-65c799fd77-b2x7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Nov 03 02:02:28.794637 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:28.794461 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-b2x7h" podUID="8d8023ca-6b91-43aa-b753-9ff28068ca28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Nov 03 02:02:29.794445 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:29.794407 5437 patch_prober.go:29] interesting pod/e2e-worker-w2-9da431-openshift-default-65c799fd77-b2x7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Nov 03 02:02:29.794959 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:29.794467 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-b2x7h" podUID="8d8023ca-6b91-43aa-b753-9ff28068ca28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Nov 03 02:02:30.641994 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:30.641954 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:02:30.897808 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:30.897759 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:02:31.084180 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:31.084143 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-b2x7h" Nov 03 02:02:31.084787 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:31.084754 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-b2x7h" Nov 03 02:02:31.085712 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:31.085691 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-b2x7h" Nov 03 02:02:33.117010 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:33.116974 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:02:33.826071 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:33.826029 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:02:34.078478 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:34.078349 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:02:34.228549 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:34.228508 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-2wlgk"] Nov 03 02:02:34.229097 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:34.229075 5437 cpu_manager.go:414] "containerMap: removing container" podUID="56f66e18-88ac-4524-aee2-ebec8cce6dd8" containerName="cleanup" Nov 03 02:02:34.229097 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:34.229095 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56f66e18-88ac-4524-aee2-ebec8cce6dd8" containerName="cleanup" Nov 03 02:02:34.229217 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:34.229107 5437 cpu_manager.go:414] "containerMap: removing container" podUID="cb08db38-b50e-4488-ada1-ffce3cf76fd1" containerName="cleanup" Nov 03 02:02:34.229217 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:34.229113 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb08db38-b50e-4488-ada1-ffce3cf76fd1" containerName="cleanup" Nov 03 02:02:34.229217 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:34.229139 5437 cpu_manager.go:414] "containerMap: removing container" podUID="653ed959-fdf9-4ad7-90be-0f58348c80a3" containerName="payload-processing" Nov 03 02:02:34.229217 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:34.229147 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="653ed959-fdf9-4ad7-90be-0f58348c80a3" containerName="payload-processing" Nov 03 02:02:34.229468 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:34.229232 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="a06e089e-fba5-403f-9fe5-34175c6865fa" containerName="istio-proxy" Nov 03 02:02:34.229468 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:34.229246 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="653ed959-fdf9-4ad7-90be-0f58348c80a3" containerName="payload-processing" Nov 03 02:02:34.229468 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:34.229255 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="56f66e18-88ac-4524-aee2-ebec8cce6dd8" containerName="cleanup" Nov 03 02:02:34.231810 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:34.231786 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-2wlgk" Nov 03 02:02:34.252076 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:34.252041 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84890f73-4e19-4ce2-a5f6-76c9fc295f02-istio-token\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-2wlgk\" (UID: \"84890f73-4e19-4ce2-a5f6-76c9fc295f02\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-2wlgk" Nov 03 02:02:34.252269 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:34.252114 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kgm5\" (UniqueName: \"kubernetes.io/projected/84890f73-4e19-4ce2-a5f6-76c9fc295f02-kube-api-access-8kgm5\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-2wlgk\" (UID: \"84890f73-4e19-4ce2-a5f6-76c9fc295f02\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-2wlgk" Nov 03 02:02:34.252269 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:34.252189 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84890f73-4e19-4ce2-a5f6-76c9fc295f02-istio-envoy\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-2wlgk\" (UID: \"84890f73-4e19-4ce2-a5f6-76c9fc295f02\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-2wlgk" Nov 03 02:02:34.252269 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:34.252240 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84890f73-4e19-4ce2-a5f6-76c9fc295f02-credential-socket\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-2wlgk\" (UID: \"84890f73-4e19-4ce2-a5f6-76c9fc295f02\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-2wlgk" Nov 03 02:02:34.252433 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:34.252301 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84890f73-4e19-4ce2-a5f6-76c9fc295f02-istio-data\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-2wlgk\" (UID: \"84890f73-4e19-4ce2-a5f6-76c9fc295f02\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-2wlgk" Nov 03 02:02:34.252433 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:34.252334 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84890f73-4e19-4ce2-a5f6-76c9fc295f02-istio-podinfo\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-2wlgk\" (UID: \"84890f73-4e19-4ce2-a5f6-76c9fc295f02\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-2wlgk" Nov 03 02:02:34.252508 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:34.252431 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84890f73-4e19-4ce2-a5f6-76c9fc295f02-workload-socket\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-2wlgk\" (UID: \"84890f73-4e19-4ce2-a5f6-76c9fc295f02\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-2wlgk" Nov 03 02:02:34.252508 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:34.252496 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84890f73-4e19-4ce2-a5f6-76c9fc295f02-workload-certs\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-2wlgk\" (UID: \"84890f73-4e19-4ce2-a5f6-76c9fc295f02\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-2wlgk" Nov 03 02:02:34.252582 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:34.252570 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84890f73-4e19-4ce2-a5f6-76c9fc295f02-istiod-ca-cert\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-2wlgk\" (UID: \"84890f73-4e19-4ce2-a5f6-76c9fc295f02\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-2wlgk" Nov 03 02:02:34.348532 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:34.348437 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-0bf34a-openshift-default-dockercfg-s9n2w\"" Nov 03 02:02:34.350852 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:34.350823 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-2wlgk"] Nov 03 02:02:34.353281 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:34.353255 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84890f73-4e19-4ce2-a5f6-76c9fc295f02-credential-socket\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-2wlgk\" (UID: \"84890f73-4e19-4ce2-a5f6-76c9fc295f02\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-2wlgk" Nov 03 02:02:34.353448 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:34.353291 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84890f73-4e19-4ce2-a5f6-76c9fc295f02-istio-data\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-2wlgk\" (UID: \"84890f73-4e19-4ce2-a5f6-76c9fc295f02\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-2wlgk" Nov 03 02:02:34.353448 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:34.353310 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84890f73-4e19-4ce2-a5f6-76c9fc295f02-istio-podinfo\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-2wlgk\" (UID: \"84890f73-4e19-4ce2-a5f6-76c9fc295f02\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-2wlgk" Nov 03 02:02:34.353448 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:34.353419 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84890f73-4e19-4ce2-a5f6-76c9fc295f02-workload-socket\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-2wlgk\" (UID: \"84890f73-4e19-4ce2-a5f6-76c9fc295f02\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-2wlgk" Nov 03 02:02:34.353646 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:34.353484 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84890f73-4e19-4ce2-a5f6-76c9fc295f02-workload-certs\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-2wlgk\" (UID: \"84890f73-4e19-4ce2-a5f6-76c9fc295f02\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-2wlgk" Nov 03 02:02:34.353646 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:34.353561 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84890f73-4e19-4ce2-a5f6-76c9fc295f02-istiod-ca-cert\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-2wlgk\" (UID: \"84890f73-4e19-4ce2-a5f6-76c9fc295f02\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-2wlgk" Nov 03 02:02:34.353646 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:34.353597 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84890f73-4e19-4ce2-a5f6-76c9fc295f02-istio-token\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-2wlgk\" (UID: \"84890f73-4e19-4ce2-a5f6-76c9fc295f02\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-2wlgk" Nov 03 02:02:34.353807 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:34.353657 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kgm5\" (UniqueName: \"kubernetes.io/projected/84890f73-4e19-4ce2-a5f6-76c9fc295f02-kube-api-access-8kgm5\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-2wlgk\" (UID: \"84890f73-4e19-4ce2-a5f6-76c9fc295f02\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-2wlgk" Nov 03 02:02:34.353807 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:34.353688 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84890f73-4e19-4ce2-a5f6-76c9fc295f02-istio-envoy\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-2wlgk\" (UID: \"84890f73-4e19-4ce2-a5f6-76c9fc295f02\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-2wlgk" Nov 03 02:02:34.353807 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:34.353756 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84890f73-4e19-4ce2-a5f6-76c9fc295f02-credential-socket\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-2wlgk\" (UID: \"84890f73-4e19-4ce2-a5f6-76c9fc295f02\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-2wlgk" Nov 03 02:02:34.354194 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:34.353850 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84890f73-4e19-4ce2-a5f6-76c9fc295f02-istio-data\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-2wlgk\" (UID: \"84890f73-4e19-4ce2-a5f6-76c9fc295f02\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-2wlgk" Nov 03 02:02:34.354194 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:34.354056 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84890f73-4e19-4ce2-a5f6-76c9fc295f02-workload-certs\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-2wlgk\" (UID: \"84890f73-4e19-4ce2-a5f6-76c9fc295f02\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-2wlgk" Nov 03 02:02:34.354325 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:34.354297 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84890f73-4e19-4ce2-a5f6-76c9fc295f02-workload-socket\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-2wlgk\" (UID: \"84890f73-4e19-4ce2-a5f6-76c9fc295f02\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-2wlgk" Nov 03 02:02:34.354670 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:34.354616 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84890f73-4e19-4ce2-a5f6-76c9fc295f02-istiod-ca-cert\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-2wlgk\" (UID: \"84890f73-4e19-4ce2-a5f6-76c9fc295f02\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-2wlgk" Nov 03 02:02:34.356536 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:34.356499 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84890f73-4e19-4ce2-a5f6-76c9fc295f02-istio-envoy\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-2wlgk\" (UID: \"84890f73-4e19-4ce2-a5f6-76c9fc295f02\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-2wlgk" Nov 03 02:02:34.356676 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:34.356660 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84890f73-4e19-4ce2-a5f6-76c9fc295f02-istio-podinfo\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-2wlgk\" (UID: \"84890f73-4e19-4ce2-a5f6-76c9fc295f02\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-2wlgk" Nov 03 02:02:34.416950 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:34.416913 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kgm5\" (UniqueName: \"kubernetes.io/projected/84890f73-4e19-4ce2-a5f6-76c9fc295f02-kube-api-access-8kgm5\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-2wlgk\" (UID: \"84890f73-4e19-4ce2-a5f6-76c9fc295f02\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-2wlgk" Nov 03 02:02:34.417341 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:34.417317 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84890f73-4e19-4ce2-a5f6-76c9fc295f02-istio-token\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-2wlgk\" (UID: \"84890f73-4e19-4ce2-a5f6-76c9fc295f02\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-2wlgk" Nov 03 02:02:34.548174 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:34.548134 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-2wlgk" Nov 03 02:02:34.969547 ip-10-0-136-16 kubenswrapper[5437]: W1103 02:02:34.969504 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84890f73_4e19_4ce2_a5f6_76c9fc295f02.slice/crio-2950f37cf65cd148d6b4bb8deaa364287b592a7691bf6dce4fb446a9a61d551c WatchSource:0}: Error finding container 2950f37cf65cd148d6b4bb8deaa364287b592a7691bf6dce4fb446a9a61d551c: Status 404 returned error can't find the container with id 2950f37cf65cd148d6b4bb8deaa364287b592a7691bf6dce4fb446a9a61d551c Nov 03 02:02:34.971594 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:34.971556 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-2wlgk"] Nov 03 02:02:34.972238 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:34.972201 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 02:02:34.972290 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:34.972277 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 02:02:34.972324 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:34.972309 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 02:02:35.051499 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:35.051458 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-2wlgk" event={"ID":"84890f73-4e19-4ce2-a5f6-76c9fc295f02","Type":"ContainerStarted","Data":"343fd6c6588636c1f4a848f3b86966ddfb555f418b1090fe328b3545aff1495a"} Nov 03 02:02:35.051674 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:35.051539 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-2wlgk" event={"ID":"84890f73-4e19-4ce2-a5f6-76c9fc295f02","Type":"ContainerStarted","Data":"2950f37cf65cd148d6b4bb8deaa364287b592a7691bf6dce4fb446a9a61d551c"} Nov 03 02:02:35.177906 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:35.177717 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-2wlgk" podStartSLOduration=1.177694111 podStartE2EDuration="1.177694111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:02:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:02:35.175160397 +0000 UTC m=+2637.204642694" watchObservedRunningTime="2026-11-03 02:02:35.177694111 +0000 UTC m=+2637.207176407" Nov 03 02:02:35.548734 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:35.548681 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-2wlgk" Nov 03 02:02:36.550565 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:36.550529 5437 patch_prober.go:29] interesting pod/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-2wlgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Nov 03 02:02:36.550957 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:36.550583 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-2wlgk" podUID="84890f73-4e19-4ce2-a5f6-76c9fc295f02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Nov 03 02:02:37.549072 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:37.549024 5437 patch_prober.go:29] interesting pod/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-2wlgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Nov 03 02:02:37.549283 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:37.549089 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-2wlgk" podUID="84890f73-4e19-4ce2-a5f6-76c9fc295f02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Nov 03 02:02:38.137659 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:38.137616 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:02:38.549504 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:38.549465 5437 patch_prober.go:29] interesting pod/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-2wlgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Nov 03 02:02:38.549672 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:38.549529 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-2wlgk" podUID="84890f73-4e19-4ce2-a5f6-76c9fc295f02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Nov 03 02:02:39.548526 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:39.548489 5437 patch_prober.go:29] interesting pod/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-2wlgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Nov 03 02:02:39.548945 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:39.548554 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-2wlgk" podUID="84890f73-4e19-4ce2-a5f6-76c9fc295f02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Nov 03 02:02:40.548999 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:40.548956 5437 patch_prober.go:29] interesting pod/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-2wlgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Nov 03 02:02:40.549446 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:40.549023 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-2wlgk" podUID="84890f73-4e19-4ce2-a5f6-76c9fc295f02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Nov 03 02:02:41.549291 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:41.549253 5437 patch_prober.go:29] interesting pod/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-2wlgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Nov 03 02:02:41.549753 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:41.549312 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-2wlgk" podUID="84890f73-4e19-4ce2-a5f6-76c9fc295f02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Nov 03 02:02:42.553817 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:42.553782 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-2wlgk" Nov 03 02:02:42.554304 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:42.554200 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-2wlgk" Nov 03 02:02:42.554838 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:42.554817 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-2wlgk" Nov 03 02:02:42.979022 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:42.978931 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:02:44.036436 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:02:44.036398 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 02:02:47.537879 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:47.537839 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:02:48.983617 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:48.983570 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:02:50.805997 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:50.805959 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:02:51.063263 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:51.063177 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:02:52.354298 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:52.354261 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:02:52.933291 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:52.933256 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-e0907d-openshift-default-59786747c7-479hs"] Nov 03 02:02:52.933564 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:52.933539 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-e0907d-openshift-default-59786747c7-479hs" podUID="1ce8bfdb-a81b-4641-996a-c5fe222b1e8c" containerName="istio-proxy" containerID="cri-o://0027529b6afedab89964a2812a63b6aad847fe93416d6ca4dd91a75a4c14bc54" gracePeriod=30 Nov 03 02:02:52.957916 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:52.957875 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:02:53.442984 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:53.442940 5437 patch_prober.go:29] interesting pod/e2e-praxis-b-e0907d-openshift-default-59786747c7-479hs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 02:02:53.443504 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:53.443006 5437 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-e0907d-openshift-default-59786747c7-479hs" podUID="1ce8bfdb-a81b-4641-996a-c5fe222b1e8c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 02:02:53.974262 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:53.974206 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:02:56.411528 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:56.411477 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:02:58.186846 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:58.186810 5437 generic.go:356] "Generic (PLEG): container finished" podID="1ce8bfdb-a81b-4641-996a-c5fe222b1e8c" containerID="0027529b6afedab89964a2812a63b6aad847fe93416d6ca4dd91a75a4c14bc54" exitCode=0 Nov 03 02:02:58.187201 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:58.186886 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-e0907d-openshift-default-59786747c7-479hs" event={"ID":"1ce8bfdb-a81b-4641-996a-c5fe222b1e8c","Type":"ContainerDied","Data":"0027529b6afedab89964a2812a63b6aad847fe93416d6ca4dd91a75a4c14bc54"} Nov 03 02:02:58.187201 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:58.186946 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-e0907d-openshift-default-59786747c7-479hs" event={"ID":"1ce8bfdb-a81b-4641-996a-c5fe222b1e8c","Type":"ContainerDied","Data":"5fd92ccbc98b7db30084da5597e42c4fedcfe0076e8d61823c8f982ec352a3b4"} Nov 03 02:02:58.187201 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:58.186970 5437 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5fd92ccbc98b7db30084da5597e42c4fedcfe0076e8d61823c8f982ec352a3b4" Nov 03 02:02:58.187201 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:58.186985 5437 scope.go:122] "RemoveContainer" containerID="d46029c563cc8d26b81544758389e98c18dbc749005e5d7ec5a52db5cd033d08" Nov 03 02:02:58.199295 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:58.199275 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-e0907d-openshift-default-59786747c7-479hs" Nov 03 02:02:58.264017 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:58.263981 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ce8bfdb-a81b-4641-996a-c5fe222b1e8c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ce8bfdb-a81b-4641-996a-c5fe222b1e8c-credential-socket\") pod \"1ce8bfdb-a81b-4641-996a-c5fe222b1e8c\" (UID: \"1ce8bfdb-a81b-4641-996a-c5fe222b1e8c\") " Nov 03 02:02:58.264228 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:58.264052 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ce8bfdb-a81b-4641-996a-c5fe222b1e8c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ce8bfdb-a81b-4641-996a-c5fe222b1e8c-workload-certs\") pod \"1ce8bfdb-a81b-4641-996a-c5fe222b1e8c\" (UID: \"1ce8bfdb-a81b-4641-996a-c5fe222b1e8c\") " Nov 03 02:02:58.264228 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:58.264092 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ce8bfdb-a81b-4641-996a-c5fe222b1e8c-istio-token\" (UniqueName: \"kubernetes.io/projected/1ce8bfdb-a81b-4641-996a-c5fe222b1e8c-istio-token\") pod \"1ce8bfdb-a81b-4641-996a-c5fe222b1e8c\" (UID: \"1ce8bfdb-a81b-4641-996a-c5fe222b1e8c\") " Nov 03 02:02:58.264228 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:58.264133 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1ce8bfdb-a81b-4641-996a-c5fe222b1e8c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ce8bfdb-a81b-4641-996a-c5fe222b1e8c-istio-podinfo\") pod \"1ce8bfdb-a81b-4641-996a-c5fe222b1e8c\" (UID: \"1ce8bfdb-a81b-4641-996a-c5fe222b1e8c\") " Nov 03 02:02:58.264228 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:58.264172 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ce8bfdb-a81b-4641-996a-c5fe222b1e8c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ce8bfdb-a81b-4641-996a-c5fe222b1e8c-istiod-ca-cert\") pod \"1ce8bfdb-a81b-4641-996a-c5fe222b1e8c\" (UID: \"1ce8bfdb-a81b-4641-996a-c5fe222b1e8c\") " Nov 03 02:02:58.264480 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:58.264254 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ce8bfdb-a81b-4641-996a-c5fe222b1e8c-credential-socket" pod "1ce8bfdb-a81b-4641-996a-c5fe222b1e8c" (UID: "1ce8bfdb-a81b-4641-996a-c5fe222b1e8c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:02:58.264480 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:58.264293 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ce8bfdb-a81b-4641-996a-c5fe222b1e8c-kube-api-access-z24pd\" (UniqueName: \"kubernetes.io/projected/1ce8bfdb-a81b-4641-996a-c5fe222b1e8c-kube-api-access-z24pd\") pod \"1ce8bfdb-a81b-4641-996a-c5fe222b1e8c\" (UID: \"1ce8bfdb-a81b-4641-996a-c5fe222b1e8c\") " Nov 03 02:02:58.264480 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:58.264357 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ce8bfdb-a81b-4641-996a-c5fe222b1e8c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ce8bfdb-a81b-4641-996a-c5fe222b1e8c-istio-data\") pod \"1ce8bfdb-a81b-4641-996a-c5fe222b1e8c\" (UID: \"1ce8bfdb-a81b-4641-996a-c5fe222b1e8c\") " Nov 03 02:02:58.264480 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:58.264391 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ce8bfdb-a81b-4641-996a-c5fe222b1e8c-workload-certs" pod "1ce8bfdb-a81b-4641-996a-c5fe222b1e8c" (UID: "1ce8bfdb-a81b-4641-996a-c5fe222b1e8c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:02:58.264480 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:58.264425 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ce8bfdb-a81b-4641-996a-c5fe222b1e8c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ce8bfdb-a81b-4641-996a-c5fe222b1e8c-istio-envoy\") pod \"1ce8bfdb-a81b-4641-996a-c5fe222b1e8c\" (UID: \"1ce8bfdb-a81b-4641-996a-c5fe222b1e8c\") " Nov 03 02:02:58.264480 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:58.264474 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ce8bfdb-a81b-4641-996a-c5fe222b1e8c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ce8bfdb-a81b-4641-996a-c5fe222b1e8c-workload-socket\") pod \"1ce8bfdb-a81b-4641-996a-c5fe222b1e8c\" (UID: \"1ce8bfdb-a81b-4641-996a-c5fe222b1e8c\") " Nov 03 02:02:58.265023 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:58.264974 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ce8bfdb-a81b-4641-996a-c5fe222b1e8c-istio-data" pod "1ce8bfdb-a81b-4641-996a-c5fe222b1e8c" (UID: "1ce8bfdb-a81b-4641-996a-c5fe222b1e8c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:02:58.265302 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:58.265276 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ce8bfdb-a81b-4641-996a-c5fe222b1e8c-workload-socket" pod "1ce8bfdb-a81b-4641-996a-c5fe222b1e8c" (UID: "1ce8bfdb-a81b-4641-996a-c5fe222b1e8c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:02:58.265508 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:58.265484 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ce8bfdb-a81b-4641-996a-c5fe222b1e8c-istiod-ca-cert" pod "1ce8bfdb-a81b-4641-996a-c5fe222b1e8c" (UID: "1ce8bfdb-a81b-4641-996a-c5fe222b1e8c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:02:58.265789 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:58.265770 5437 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ce8bfdb-a81b-4641-996a-c5fe222b1e8c-workload-certs\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:02:58.265882 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:58.265795 5437 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ce8bfdb-a81b-4641-996a-c5fe222b1e8c-istiod-ca-cert\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:02:58.265882 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:58.265810 5437 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ce8bfdb-a81b-4641-996a-c5fe222b1e8c-istio-data\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:02:58.265882 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:58.265824 5437 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ce8bfdb-a81b-4641-996a-c5fe222b1e8c-workload-socket\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:02:58.265882 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:58.265838 5437 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ce8bfdb-a81b-4641-996a-c5fe222b1e8c-credential-socket\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:02:58.267612 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:58.266810 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ce8bfdb-a81b-4641-996a-c5fe222b1e8c-istio-token" pod "1ce8bfdb-a81b-4641-996a-c5fe222b1e8c" (UID: "1ce8bfdb-a81b-4641-996a-c5fe222b1e8c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:02:58.267612 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:58.266939 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ce8bfdb-a81b-4641-996a-c5fe222b1e8c-kube-api-access-z24pd" pod "1ce8bfdb-a81b-4641-996a-c5fe222b1e8c" (UID: "1ce8bfdb-a81b-4641-996a-c5fe222b1e8c"). InnerVolumeSpecName "kube-api-access-z24pd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:02:58.267612 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:58.266988 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ce8bfdb-a81b-4641-996a-c5fe222b1e8c-istio-envoy" pod "1ce8bfdb-a81b-4641-996a-c5fe222b1e8c" (UID: "1ce8bfdb-a81b-4641-996a-c5fe222b1e8c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:02:58.267864 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:58.267846 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1ce8bfdb-a81b-4641-996a-c5fe222b1e8c-istio-podinfo" pod "1ce8bfdb-a81b-4641-996a-c5fe222b1e8c" (UID: "1ce8bfdb-a81b-4641-996a-c5fe222b1e8c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:02:58.366667 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:58.366633 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z24pd\" (UniqueName: \"kubernetes.io/projected/1ce8bfdb-a81b-4641-996a-c5fe222b1e8c-kube-api-access-z24pd\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:02:58.366667 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:58.366662 5437 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ce8bfdb-a81b-4641-996a-c5fe222b1e8c-istio-envoy\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:02:58.366667 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:58.366673 5437 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ce8bfdb-a81b-4641-996a-c5fe222b1e8c-istio-token\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:02:58.366913 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:58.366681 5437 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ce8bfdb-a81b-4641-996a-c5fe222b1e8c-istio-podinfo\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:02:58.980606 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:58.980558 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:02:59.193718 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:59.193688 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-e0907d-openshift-default-59786747c7-479hs" Nov 03 02:02:59.488886 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:59.488846 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-e0907d-openshift-default-59786747c7-479hs"] Nov 03 02:02:59.491849 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:02:59.491813 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-e0907d-openshift-default-59786747c7-479hs"] Nov 03 02:03:00.089119 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:00.089084 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ce8bfdb-a81b-4641-996a-c5fe222b1e8c" path="/var/lib/kubelet/pods/1ce8bfdb-a81b-4641-996a-c5fe222b1e8c/volumes" Nov 03 02:03:03.614775 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:03.614740 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:03:03.843717 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:03.843649 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:03:08.618126 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:08.618088 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:03:12.598280 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:12.598239 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:03:13.388839 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:13.388804 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:03:14.035456 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:03:14.035417 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 02:03:18.169813 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:18.169768 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:03:22.924643 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:22.924602 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-ngbgz"] Nov 03 02:03:22.925215 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:22.925198 5437 cpu_manager.go:414] "containerMap: removing container" podUID="1ce8bfdb-a81b-4641-996a-c5fe222b1e8c" containerName="istio-proxy" Nov 03 02:03:22.925260 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:22.925216 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ce8bfdb-a81b-4641-996a-c5fe222b1e8c" containerName="istio-proxy" Nov 03 02:03:22.925260 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:22.925235 5437 cpu_manager.go:414] "containerMap: removing container" podUID="1ce8bfdb-a81b-4641-996a-c5fe222b1e8c" containerName="istio-proxy" Nov 03 02:03:22.925260 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:22.925241 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ce8bfdb-a81b-4641-996a-c5fe222b1e8c" containerName="istio-proxy" Nov 03 02:03:22.925355 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:22.925341 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ce8bfdb-a81b-4641-996a-c5fe222b1e8c" containerName="istio-proxy" Nov 03 02:03:22.925355 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:22.925353 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ce8bfdb-a81b-4641-996a-c5fe222b1e8c" containerName="istio-proxy" Nov 03 02:03:22.928849 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:22.928825 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-ngbgz" Nov 03 02:03:22.981611 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:22.981574 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:03:23.038073 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:23.038034 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-ngbgz"] Nov 03 02:03:23.043636 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:23.043599 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2740d82-3082-474a-8726-4aaaecf57272-istio-data\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-ngbgz\" (UID: \"b2740d82-3082-474a-8726-4aaaecf57272\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-ngbgz" Nov 03 02:03:23.043803 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:23.043655 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2740d82-3082-474a-8726-4aaaecf57272-credential-socket\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-ngbgz\" (UID: \"b2740d82-3082-474a-8726-4aaaecf57272\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-ngbgz" Nov 03 02:03:23.043803 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:23.043789 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2740d82-3082-474a-8726-4aaaecf57272-workload-socket\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-ngbgz\" (UID: \"b2740d82-3082-474a-8726-4aaaecf57272\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-ngbgz" Nov 03 02:03:23.043875 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:23.043831 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krz6h\" (UniqueName: \"kubernetes.io/projected/b2740d82-3082-474a-8726-4aaaecf57272-kube-api-access-krz6h\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-ngbgz\" (UID: \"b2740d82-3082-474a-8726-4aaaecf57272\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-ngbgz" Nov 03 02:03:23.043983 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:23.043964 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2740d82-3082-474a-8726-4aaaecf57272-workload-certs\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-ngbgz\" (UID: \"b2740d82-3082-474a-8726-4aaaecf57272\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-ngbgz" Nov 03 02:03:23.044018 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:23.044008 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2740d82-3082-474a-8726-4aaaecf57272-istio-envoy\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-ngbgz\" (UID: \"b2740d82-3082-474a-8726-4aaaecf57272\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-ngbgz" Nov 03 02:03:23.044120 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:23.044097 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2740d82-3082-474a-8726-4aaaecf57272-istio-token\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-ngbgz\" (UID: \"b2740d82-3082-474a-8726-4aaaecf57272\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-ngbgz" Nov 03 02:03:23.044171 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:23.044158 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2740d82-3082-474a-8726-4aaaecf57272-istio-podinfo\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-ngbgz\" (UID: \"b2740d82-3082-474a-8726-4aaaecf57272\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-ngbgz" Nov 03 02:03:23.044261 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:23.044233 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2740d82-3082-474a-8726-4aaaecf57272-istiod-ca-cert\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-ngbgz\" (UID: \"b2740d82-3082-474a-8726-4aaaecf57272\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-ngbgz" Nov 03 02:03:23.145949 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:23.145901 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krz6h\" (UniqueName: \"kubernetes.io/projected/b2740d82-3082-474a-8726-4aaaecf57272-kube-api-access-krz6h\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-ngbgz\" (UID: \"b2740d82-3082-474a-8726-4aaaecf57272\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-ngbgz" Nov 03 02:03:23.146174 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:23.146018 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2740d82-3082-474a-8726-4aaaecf57272-workload-certs\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-ngbgz\" (UID: \"b2740d82-3082-474a-8726-4aaaecf57272\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-ngbgz" Nov 03 02:03:23.146174 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:23.146077 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2740d82-3082-474a-8726-4aaaecf57272-istio-envoy\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-ngbgz\" (UID: \"b2740d82-3082-474a-8726-4aaaecf57272\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-ngbgz" Nov 03 02:03:23.146174 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:23.146134 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2740d82-3082-474a-8726-4aaaecf57272-istio-token\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-ngbgz\" (UID: \"b2740d82-3082-474a-8726-4aaaecf57272\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-ngbgz" Nov 03 02:03:23.146353 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:23.146195 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2740d82-3082-474a-8726-4aaaecf57272-istio-podinfo\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-ngbgz\" (UID: \"b2740d82-3082-474a-8726-4aaaecf57272\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-ngbgz" Nov 03 02:03:23.146353 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:23.146219 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2740d82-3082-474a-8726-4aaaecf57272-istiod-ca-cert\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-ngbgz\" (UID: \"b2740d82-3082-474a-8726-4aaaecf57272\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-ngbgz" Nov 03 02:03:23.146353 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:23.146282 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2740d82-3082-474a-8726-4aaaecf57272-istio-data\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-ngbgz\" (UID: \"b2740d82-3082-474a-8726-4aaaecf57272\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-ngbgz" Nov 03 02:03:23.146353 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:23.146316 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2740d82-3082-474a-8726-4aaaecf57272-credential-socket\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-ngbgz\" (UID: \"b2740d82-3082-474a-8726-4aaaecf57272\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-ngbgz" Nov 03 02:03:23.146591 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:23.146356 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2740d82-3082-474a-8726-4aaaecf57272-workload-socket\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-ngbgz\" (UID: \"b2740d82-3082-474a-8726-4aaaecf57272\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-ngbgz" Nov 03 02:03:23.146591 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:23.146551 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2740d82-3082-474a-8726-4aaaecf57272-workload-certs\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-ngbgz\" (UID: \"b2740d82-3082-474a-8726-4aaaecf57272\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-ngbgz" Nov 03 02:03:23.146701 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:23.146675 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2740d82-3082-474a-8726-4aaaecf57272-istio-data\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-ngbgz\" (UID: \"b2740d82-3082-474a-8726-4aaaecf57272\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-ngbgz" Nov 03 02:03:23.146744 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:23.146729 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2740d82-3082-474a-8726-4aaaecf57272-workload-socket\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-ngbgz\" (UID: \"b2740d82-3082-474a-8726-4aaaecf57272\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-ngbgz" Nov 03 02:03:23.146803 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:23.146776 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2740d82-3082-474a-8726-4aaaecf57272-credential-socket\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-ngbgz\" (UID: \"b2740d82-3082-474a-8726-4aaaecf57272\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-ngbgz" Nov 03 02:03:23.147308 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:23.147287 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2740d82-3082-474a-8726-4aaaecf57272-istiod-ca-cert\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-ngbgz\" (UID: \"b2740d82-3082-474a-8726-4aaaecf57272\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-ngbgz" Nov 03 02:03:23.148672 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:23.148649 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2740d82-3082-474a-8726-4aaaecf57272-istio-envoy\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-ngbgz\" (UID: \"b2740d82-3082-474a-8726-4aaaecf57272\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-ngbgz" Nov 03 02:03:23.148828 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:23.148716 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2740d82-3082-474a-8726-4aaaecf57272-istio-podinfo\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-ngbgz\" (UID: \"b2740d82-3082-474a-8726-4aaaecf57272\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-ngbgz" Nov 03 02:03:23.209707 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:23.209637 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2740d82-3082-474a-8726-4aaaecf57272-istio-token\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-ngbgz\" (UID: \"b2740d82-3082-474a-8726-4aaaecf57272\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-ngbgz" Nov 03 02:03:23.209707 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:23.209645 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krz6h\" (UniqueName: \"kubernetes.io/projected/b2740d82-3082-474a-8726-4aaaecf57272-kube-api-access-krz6h\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-ngbgz\" (UID: \"b2740d82-3082-474a-8726-4aaaecf57272\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-ngbgz" Nov 03 02:03:23.242616 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:23.242578 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-ngbgz" Nov 03 02:03:23.667559 ip-10-0-136-16 kubenswrapper[5437]: W1103 02:03:23.667518 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2740d82_3082_474a_8726_4aaaecf57272.slice/crio-622b2b8f64364e1d2231e53585a67424315a4e8d40e424ed75ad37996b6bdc66 WatchSource:0}: Error finding container 622b2b8f64364e1d2231e53585a67424315a4e8d40e424ed75ad37996b6bdc66: Status 404 returned error can't find the container with id 622b2b8f64364e1d2231e53585a67424315a4e8d40e424ed75ad37996b6bdc66 Nov 03 02:03:23.670064 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:23.670029 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-ngbgz"] Nov 03 02:03:23.670516 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:23.670480 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 02:03:23.670621 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:23.670568 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 02:03:23.670689 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:23.670616 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 02:03:24.339761 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:24.339719 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-ngbgz" event={"ID":"b2740d82-3082-474a-8726-4aaaecf57272","Type":"ContainerStarted","Data":"875333aad582145899d9f30bb4e88f15e8d54be9541e1f768dc8ebd1a69ff108"} Nov 03 02:03:24.340187 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:24.339790 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-ngbgz" event={"ID":"b2740d82-3082-474a-8726-4aaaecf57272","Type":"ContainerStarted","Data":"622b2b8f64364e1d2231e53585a67424315a4e8d40e424ed75ad37996b6bdc66"} Nov 03 02:03:24.463640 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:24.463586 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-ngbgz" podStartSLOduration=2.463572659 podStartE2EDuration="2.463572659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:03:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:03:24.462533174 +0000 UTC m=+2686.492015470" watchObservedRunningTime="2026-11-03 02:03:24.463572659 +0000 UTC m=+2686.493054953" Nov 03 02:03:25.243426 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:25.243380 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-ngbgz" Nov 03 02:03:25.245471 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:25.245434 5437 patch_prober.go:29] interesting pod/e2e-worker-w2-9da431-openshift-default-65c799fd77-ngbgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body= Nov 03 02:03:25.245724 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:25.245495 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-ngbgz" podUID="b2740d82-3082-474a-8726-4aaaecf57272" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" Nov 03 02:03:26.243909 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:26.243869 5437 patch_prober.go:29] interesting pod/e2e-worker-w2-9da431-openshift-default-65c799fd77-ngbgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body= Nov 03 02:03:26.244332 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:26.243940 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-ngbgz" podUID="b2740d82-3082-474a-8726-4aaaecf57272" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" Nov 03 02:03:27.243041 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:27.242999 5437 patch_prober.go:29] interesting pod/e2e-worker-w2-9da431-openshift-default-65c799fd77-ngbgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body= Nov 03 02:03:27.243229 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:27.243060 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-ngbgz" podUID="b2740d82-3082-474a-8726-4aaaecf57272" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" Nov 03 02:03:27.834387 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:27.833693 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:03:28.248452 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:28.247809 5437 patch_prober.go:29] interesting pod/e2e-worker-w2-9da431-openshift-default-65c799fd77-ngbgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body= Nov 03 02:03:28.248452 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:28.247885 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-ngbgz" podUID="b2740d82-3082-474a-8726-4aaaecf57272" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" Nov 03 02:03:29.248395 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:29.247082 5437 patch_prober.go:29] interesting pod/e2e-worker-w2-9da431-openshift-default-65c799fd77-ngbgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body= Nov 03 02:03:29.248395 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:29.247151 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-ngbgz" podUID="b2740d82-3082-474a-8726-4aaaecf57272" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" Nov 03 02:03:30.243525 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:30.243470 5437 patch_prober.go:29] interesting pod/e2e-worker-w2-9da431-openshift-default-65c799fd77-ngbgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body= Nov 03 02:03:30.243714 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:30.243597 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-ngbgz" podUID="b2740d82-3082-474a-8726-4aaaecf57272" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" Nov 03 02:03:31.632547 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:31.632511 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-ngbgz" Nov 03 02:03:31.633014 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:31.632934 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-ngbgz" Nov 03 02:03:31.633637 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:31.633616 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-ngbgz" Nov 03 02:03:32.707578 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:32.707541 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:03:37.491704 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:37.491651 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:03:42.296526 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:42.296481 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:03:44.036807 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:03:44.036764 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 02:03:45.330592 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:45.330553 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:03:46.001676 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:46.001636 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:03:46.264053 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:46.263954 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:03:47.282640 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:47.282549 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:03:47.483296 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:47.483246 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:03:48.644474 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:48.644400 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:03:49.196525 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:49.196486 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:03:49.211211 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:49.211178 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-8hvq2"] Nov 03 02:03:49.215482 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:49.215449 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-8hvq2" Nov 03 02:03:49.314648 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:49.311663 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-8hvq2"] Nov 03 02:03:49.392530 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:49.392489 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d287f009-3621-4858-9eed-5b806f637d6b-workload-socket\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-8hvq2\" (UID: \"d287f009-3621-4858-9eed-5b806f637d6b\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-8hvq2" Nov 03 02:03:49.392734 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:49.392538 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d287f009-3621-4858-9eed-5b806f637d6b-istio-token\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-8hvq2\" (UID: \"d287f009-3621-4858-9eed-5b806f637d6b\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-8hvq2" Nov 03 02:03:49.392734 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:49.392683 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d287f009-3621-4858-9eed-5b806f637d6b-istio-envoy\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-8hvq2\" (UID: \"d287f009-3621-4858-9eed-5b806f637d6b\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-8hvq2" Nov 03 02:03:49.392734 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:49.392715 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d287f009-3621-4858-9eed-5b806f637d6b-istiod-ca-cert\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-8hvq2\" (UID: \"d287f009-3621-4858-9eed-5b806f637d6b\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-8hvq2" Nov 03 02:03:49.392840 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:49.392768 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d287f009-3621-4858-9eed-5b806f637d6b-credential-socket\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-8hvq2\" (UID: \"d287f009-3621-4858-9eed-5b806f637d6b\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-8hvq2" Nov 03 02:03:49.392875 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:49.392838 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d287f009-3621-4858-9eed-5b806f637d6b-workload-certs\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-8hvq2\" (UID: \"d287f009-3621-4858-9eed-5b806f637d6b\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-8hvq2" Nov 03 02:03:49.392910 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:49.392876 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d287f009-3621-4858-9eed-5b806f637d6b-istio-podinfo\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-8hvq2\" (UID: \"d287f009-3621-4858-9eed-5b806f637d6b\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-8hvq2" Nov 03 02:03:49.392910 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:49.392897 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d287f009-3621-4858-9eed-5b806f637d6b-istio-data\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-8hvq2\" (UID: \"d287f009-3621-4858-9eed-5b806f637d6b\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-8hvq2" Nov 03 02:03:49.393038 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:49.393021 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qddk\" (UniqueName: \"kubernetes.io/projected/d287f009-3621-4858-9eed-5b806f637d6b-kube-api-access-4qddk\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-8hvq2\" (UID: \"d287f009-3621-4858-9eed-5b806f637d6b\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-8hvq2" Nov 03 02:03:49.399588 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:49.399553 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:03:49.494603 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:49.494496 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d287f009-3621-4858-9eed-5b806f637d6b-istio-envoy\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-8hvq2\" (UID: \"d287f009-3621-4858-9eed-5b806f637d6b\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-8hvq2" Nov 03 02:03:49.494603 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:49.494548 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d287f009-3621-4858-9eed-5b806f637d6b-istiod-ca-cert\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-8hvq2\" (UID: \"d287f009-3621-4858-9eed-5b806f637d6b\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-8hvq2" Nov 03 02:03:49.494865 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:49.494629 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d287f009-3621-4858-9eed-5b806f637d6b-credential-socket\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-8hvq2\" (UID: \"d287f009-3621-4858-9eed-5b806f637d6b\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-8hvq2" Nov 03 02:03:49.494865 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:49.494662 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d287f009-3621-4858-9eed-5b806f637d6b-workload-certs\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-8hvq2\" (UID: \"d287f009-3621-4858-9eed-5b806f637d6b\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-8hvq2" Nov 03 02:03:49.494865 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:49.494699 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d287f009-3621-4858-9eed-5b806f637d6b-istio-podinfo\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-8hvq2\" (UID: \"d287f009-3621-4858-9eed-5b806f637d6b\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-8hvq2" Nov 03 02:03:49.494865 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:49.494729 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d287f009-3621-4858-9eed-5b806f637d6b-istio-data\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-8hvq2\" (UID: \"d287f009-3621-4858-9eed-5b806f637d6b\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-8hvq2" Nov 03 02:03:49.494865 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:49.494827 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qddk\" (UniqueName: \"kubernetes.io/projected/d287f009-3621-4858-9eed-5b806f637d6b-kube-api-access-4qddk\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-8hvq2\" (UID: \"d287f009-3621-4858-9eed-5b806f637d6b\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-8hvq2" Nov 03 02:03:49.495132 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:49.494905 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d287f009-3621-4858-9eed-5b806f637d6b-workload-socket\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-8hvq2\" (UID: \"d287f009-3621-4858-9eed-5b806f637d6b\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-8hvq2" Nov 03 02:03:49.495132 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:49.494930 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d287f009-3621-4858-9eed-5b806f637d6b-istio-token\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-8hvq2\" (UID: \"d287f009-3621-4858-9eed-5b806f637d6b\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-8hvq2" Nov 03 02:03:49.495492 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:49.495460 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d287f009-3621-4858-9eed-5b806f637d6b-istiod-ca-cert\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-8hvq2\" (UID: \"d287f009-3621-4858-9eed-5b806f637d6b\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-8hvq2" Nov 03 02:03:49.495613 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:49.495585 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d287f009-3621-4858-9eed-5b806f637d6b-istio-data\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-8hvq2\" (UID: \"d287f009-3621-4858-9eed-5b806f637d6b\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-8hvq2" Nov 03 02:03:49.495802 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:49.495778 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d287f009-3621-4858-9eed-5b806f637d6b-credential-socket\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-8hvq2\" (UID: \"d287f009-3621-4858-9eed-5b806f637d6b\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-8hvq2" Nov 03 02:03:49.495876 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:49.495858 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d287f009-3621-4858-9eed-5b806f637d6b-workload-certs\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-8hvq2\" (UID: \"d287f009-3621-4858-9eed-5b806f637d6b\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-8hvq2" Nov 03 02:03:49.495934 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:49.495866 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d287f009-3621-4858-9eed-5b806f637d6b-workload-socket\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-8hvq2\" (UID: \"d287f009-3621-4858-9eed-5b806f637d6b\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-8hvq2" Nov 03 02:03:49.506418 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:49.497349 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d287f009-3621-4858-9eed-5b806f637d6b-istio-envoy\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-8hvq2\" (UID: \"d287f009-3621-4858-9eed-5b806f637d6b\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-8hvq2" Nov 03 02:03:49.506418 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:49.497439 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d287f009-3621-4858-9eed-5b806f637d6b-istio-podinfo\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-8hvq2\" (UID: \"d287f009-3621-4858-9eed-5b806f637d6b\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-8hvq2" Nov 03 02:03:49.558123 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:49.558081 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d287f009-3621-4858-9eed-5b806f637d6b-istio-token\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-8hvq2\" (UID: \"d287f009-3621-4858-9eed-5b806f637d6b\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-8hvq2" Nov 03 02:03:49.558287 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:49.558130 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qddk\" (UniqueName: \"kubernetes.io/projected/d287f009-3621-4858-9eed-5b806f637d6b-kube-api-access-4qddk\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-8hvq2\" (UID: \"d287f009-3621-4858-9eed-5b806f637d6b\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-8hvq2" Nov 03 02:03:49.609345 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:49.609306 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:03:49.834491 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:49.834448 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-8hvq2" Nov 03 02:03:50.259903 ip-10-0-136-16 kubenswrapper[5437]: W1103 02:03:50.259874 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd287f009_3621_4858_9eed_5b806f637d6b.slice/crio-39f73ecb17f12492717f7e1253a722df3e9b2ca109263a22eec3eac254238766 WatchSource:0}: Error finding container 39f73ecb17f12492717f7e1253a722df3e9b2ca109263a22eec3eac254238766: Status 404 returned error can't find the container with id 39f73ecb17f12492717f7e1253a722df3e9b2ca109263a22eec3eac254238766 Nov 03 02:03:50.262848 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:50.262821 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-8hvq2"] Nov 03 02:03:50.263613 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:50.263581 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 02:03:50.263683 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:50.263645 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 02:03:50.263683 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:50.263674 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 02:03:50.409176 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:50.409136 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:03:50.516188 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:50.516087 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-8hvq2" event={"ID":"d287f009-3621-4858-9eed-5b806f637d6b","Type":"ContainerStarted","Data":"3c8b1c781f4d9c03c6536661c2217530b97662d1a785f39cc23bf2241bfb7914"} Nov 03 02:03:50.516188 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:50.516158 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-8hvq2" event={"ID":"d287f009-3621-4858-9eed-5b806f637d6b","Type":"ContainerStarted","Data":"39f73ecb17f12492717f7e1253a722df3e9b2ca109263a22eec3eac254238766"} Nov 03 02:03:50.610326 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:50.610285 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:03:50.641040 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:50.640975 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-8hvq2" podStartSLOduration=1.640958773 podStartE2EDuration="1.640958773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:03:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:03:50.639479216 +0000 UTC m=+2712.668961510" watchObservedRunningTime="2026-11-03 02:03:50.640958773 +0000 UTC m=+2712.670441069" Nov 03 02:03:50.835019 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:50.834978 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-8hvq2" Nov 03 02:03:51.197903 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:51.197804 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:03:51.761933 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:51.761878 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:03:51.839324 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:51.839282 5437 patch_prober.go:29] interesting pod/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-8hvq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Nov 03 02:03:51.840029 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:51.839994 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-8hvq2" podUID="d287f009-3621-4858-9eed-5b806f637d6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Nov 03 02:03:52.384794 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:52.384755 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:03:52.598335 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:52.598295 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:03:52.806857 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:52.806811 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:03:52.836815 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:52.836773 5437 patch_prober.go:29] interesting pod/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-8hvq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Nov 03 02:03:52.837102 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:52.837072 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-8hvq2" podUID="d287f009-3621-4858-9eed-5b806f637d6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Nov 03 02:03:53.644012 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:53.643975 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:03:53.835961 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:53.835922 5437 patch_prober.go:29] interesting pod/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-8hvq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Nov 03 02:03:53.836435 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:53.836402 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-8hvq2" podUID="d287f009-3621-4858-9eed-5b806f637d6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Nov 03 02:03:53.844192 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:53.844138 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:03:54.160432 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:54.144249 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-059b23-74cdd47845-8jxpp"] Nov 03 02:03:54.169571 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:54.169526 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-059b23-74cdd47845-8jxpp" Nov 03 02:03:54.211551 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:54.211508 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-059b23-74cdd47845-8jxpp"] Nov 03 02:03:54.243872 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:54.243833 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-059b23-dockercfg-znrsf\"" Nov 03 02:03:54.244949 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:54.244922 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-059b23\"" Nov 03 02:03:54.279252 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:54.279215 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f622668a-42fc-4471-9b7f-1166924c2769-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-059b23-74cdd47845-8jxpp\" (UID: \"f622668a-42fc-4471-9b7f-1166924c2769\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-059b23-74cdd47845-8jxpp" Nov 03 02:03:54.279441 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:54.279270 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nj6bg\" (UniqueName: \"kubernetes.io/projected/f622668a-42fc-4471-9b7f-1166924c2769-kube-api-access-nj6bg\") pod \"payload-processing-e2e-shared-a-w3-059b23-74cdd47845-8jxpp\" (UID: \"f622668a-42fc-4471-9b7f-1166924c2769\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-059b23-74cdd47845-8jxpp" Nov 03 02:03:54.282773 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:54.282728 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:03:54.380225 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:54.380186 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f622668a-42fc-4471-9b7f-1166924c2769-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-059b23-74cdd47845-8jxpp\" (UID: \"f622668a-42fc-4471-9b7f-1166924c2769\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-059b23-74cdd47845-8jxpp" Nov 03 02:03:54.380456 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:54.380239 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nj6bg\" (UniqueName: \"kubernetes.io/projected/f622668a-42fc-4471-9b7f-1166924c2769-kube-api-access-nj6bg\") pod \"payload-processing-e2e-shared-a-w3-059b23-74cdd47845-8jxpp\" (UID: \"f622668a-42fc-4471-9b7f-1166924c2769\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-059b23-74cdd47845-8jxpp" Nov 03 02:03:54.380950 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:54.380929 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f622668a-42fc-4471-9b7f-1166924c2769-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-059b23-74cdd47845-8jxpp\" (UID: \"f622668a-42fc-4471-9b7f-1166924c2769\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-059b23-74cdd47845-8jxpp" Nov 03 02:03:54.446438 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:54.446048 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nj6bg\" (UniqueName: \"kubernetes.io/projected/f622668a-42fc-4471-9b7f-1166924c2769-kube-api-access-nj6bg\") pod \"payload-processing-e2e-shared-a-w3-059b23-74cdd47845-8jxpp\" (UID: \"f622668a-42fc-4471-9b7f-1166924c2769\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-059b23-74cdd47845-8jxpp" Nov 03 02:03:54.490066 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:54.490024 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-059b23-74cdd47845-8jxpp" Nov 03 02:03:54.499636 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:54.499598 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:03:54.842860 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:54.842815 5437 patch_prober.go:29] interesting pod/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-8hvq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Nov 03 02:03:54.843542 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:54.842897 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-8hvq2" podUID="d287f009-3621-4858-9eed-5b806f637d6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Nov 03 02:03:54.896640 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:54.896506 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-059b23-74cdd47845-8jxpp"] Nov 03 02:03:54.897402 ip-10-0-136-16 kubenswrapper[5437]: W1103 02:03:54.897342 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf622668a_42fc_4471_9b7f_1166924c2769.slice/crio-5b10727791f122f20b7b980abb5f2048f52166d128056c1b5103ffaabede29e9 WatchSource:0}: Error finding container 5b10727791f122f20b7b980abb5f2048f52166d128056c1b5103ffaabede29e9: Status 404 returned error can't find the container with id 5b10727791f122f20b7b980abb5f2048f52166d128056c1b5103ffaabede29e9 Nov 03 02:03:55.599856 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:55.592621 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-059b23-74cdd47845-8jxpp" event={"ID":"f622668a-42fc-4471-9b7f-1166924c2769","Type":"ContainerStarted","Data":"45c70dbd38645b993d840ab9a97d78af99dd76b4971696776eccab878efba977"} Nov 03 02:03:55.599856 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:55.592732 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-059b23-74cdd47845-8jxpp" event={"ID":"f622668a-42fc-4471-9b7f-1166924c2769","Type":"ContainerStarted","Data":"5b10727791f122f20b7b980abb5f2048f52166d128056c1b5103ffaabede29e9"} Nov 03 02:03:55.599856 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:55.593735 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-059b23-74cdd47845-8jxpp" Nov 03 02:03:55.715289 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:55.714705 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-059b23-74cdd47845-8jxpp" podStartSLOduration=1.714689387 podStartE2EDuration="1.714689387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:03:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:03:55.712561257 +0000 UTC m=+2717.742043555" watchObservedRunningTime="2026-11-03 02:03:55.714689387 +0000 UTC m=+2717.744171681" Nov 03 02:03:55.838389 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:55.837813 5437 patch_prober.go:29] interesting pod/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-8hvq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Nov 03 02:03:55.838389 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:55.837883 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-8hvq2" podUID="d287f009-3621-4858-9eed-5b806f637d6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Nov 03 02:03:55.981889 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:55.981616 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:03:56.835302 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:56.835258 5437 patch_prober.go:29] interesting pod/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-8hvq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Nov 03 02:03:56.835674 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:56.835638 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-8hvq2" podUID="d287f009-3621-4858-9eed-5b806f637d6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Nov 03 02:03:58.027229 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:58.027185 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:03:58.136001 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:58.135962 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-8hvq2" Nov 03 02:03:58.136420 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:58.136393 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-8hvq2" Nov 03 02:03:58.137424 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:58.137401 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-8hvq2" Nov 03 02:03:58.232998 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:58.232950 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:03:58.527595 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:58.527558 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:03:58.731403 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:58.731299 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:03:58.874728 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:58.874684 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-f7f61d-7cf44f54d7-xg8mf"] Nov 03 02:03:58.879258 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:58.879223 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f7f61d-7cf44f54d7-xg8mf" Nov 03 02:03:58.935974 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:58.935932 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-f7f61d-7cf44f54d7-xg8mf"] Nov 03 02:03:58.961154 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:58.960393 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-f7f61d\"" Nov 03 02:03:58.991689 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:58.990188 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11b45fe9-41b7-445c-b886-34b15a5e0d8a-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-f7f61d-7cf44f54d7-xg8mf\" (UID: \"11b45fe9-41b7-445c-b886-34b15a5e0d8a\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f7f61d-7cf44f54d7-xg8mf" Nov 03 02:03:58.991689 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:58.990514 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hskfw\" (UniqueName: \"kubernetes.io/projected/11b45fe9-41b7-445c-b886-34b15a5e0d8a-kube-api-access-hskfw\") pod \"payload-pre-processing-e2e-praxis-b-f7f61d-7cf44f54d7-xg8mf\" (UID: \"11b45fe9-41b7-445c-b886-34b15a5e0d8a\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f7f61d-7cf44f54d7-xg8mf" Nov 03 02:03:59.008830 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:59.008787 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-f7f61d-6dcb8fbbb5-n2wb4"] Nov 03 02:03:59.013672 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:59.013644 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-f7f61d-6dcb8fbbb5-n2wb4" Nov 03 02:03:59.028054 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:59.028019 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-f7f61d-dockercfg-kqffb\"" Nov 03 02:03:59.069264 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:59.069225 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-f7f61d-6dcb8fbbb5-n2wb4"] Nov 03 02:03:59.091490 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:59.091448 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0773a31e-b378-4c6a-9f35-7ba46f2e0dc7-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-f7f61d-6dcb8fbbb5-n2wb4\" (UID: \"0773a31e-b378-4c6a-9f35-7ba46f2e0dc7\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-f7f61d-6dcb8fbbb5-n2wb4" Nov 03 02:03:59.091683 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:59.091517 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hskfw\" (UniqueName: \"kubernetes.io/projected/11b45fe9-41b7-445c-b886-34b15a5e0d8a-kube-api-access-hskfw\") pod \"payload-pre-processing-e2e-praxis-b-f7f61d-7cf44f54d7-xg8mf\" (UID: \"11b45fe9-41b7-445c-b886-34b15a5e0d8a\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f7f61d-7cf44f54d7-xg8mf" Nov 03 02:03:59.091683 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:59.091599 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11b45fe9-41b7-445c-b886-34b15a5e0d8a-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-f7f61d-7cf44f54d7-xg8mf\" (UID: \"11b45fe9-41b7-445c-b886-34b15a5e0d8a\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f7f61d-7cf44f54d7-xg8mf" Nov 03 02:03:59.091827 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:59.091712 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzg2t\" (UniqueName: \"kubernetes.io/projected/0773a31e-b378-4c6a-9f35-7ba46f2e0dc7-kube-api-access-mzg2t\") pod \"payload-processing-e2e-praxis-b-f7f61d-6dcb8fbbb5-n2wb4\" (UID: \"0773a31e-b378-4c6a-9f35-7ba46f2e0dc7\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-f7f61d-6dcb8fbbb5-n2wb4" Nov 03 02:03:59.092291 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:59.092268 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11b45fe9-41b7-445c-b886-34b15a5e0d8a-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-f7f61d-7cf44f54d7-xg8mf\" (UID: \"11b45fe9-41b7-445c-b886-34b15a5e0d8a\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f7f61d-7cf44f54d7-xg8mf" Nov 03 02:03:59.163357 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:59.162622 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hskfw\" (UniqueName: \"kubernetes.io/projected/11b45fe9-41b7-445c-b886-34b15a5e0d8a-kube-api-access-hskfw\") pod \"payload-pre-processing-e2e-praxis-b-f7f61d-7cf44f54d7-xg8mf\" (UID: \"11b45fe9-41b7-445c-b886-34b15a5e0d8a\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f7f61d-7cf44f54d7-xg8mf" Nov 03 02:03:59.193179 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:59.193144 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzg2t\" (UniqueName: \"kubernetes.io/projected/0773a31e-b378-4c6a-9f35-7ba46f2e0dc7-kube-api-access-mzg2t\") pod \"payload-processing-e2e-praxis-b-f7f61d-6dcb8fbbb5-n2wb4\" (UID: \"0773a31e-b378-4c6a-9f35-7ba46f2e0dc7\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-f7f61d-6dcb8fbbb5-n2wb4" Nov 03 02:03:59.193376 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:59.193260 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0773a31e-b378-4c6a-9f35-7ba46f2e0dc7-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-f7f61d-6dcb8fbbb5-n2wb4\" (UID: \"0773a31e-b378-4c6a-9f35-7ba46f2e0dc7\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-f7f61d-6dcb8fbbb5-n2wb4" Nov 03 02:03:59.194001 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:59.193981 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0773a31e-b378-4c6a-9f35-7ba46f2e0dc7-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-f7f61d-6dcb8fbbb5-n2wb4\" (UID: \"0773a31e-b378-4c6a-9f35-7ba46f2e0dc7\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-f7f61d-6dcb8fbbb5-n2wb4" Nov 03 02:03:59.195841 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:59.195826 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f7f61d-7cf44f54d7-xg8mf" Nov 03 02:03:59.260425 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:59.257009 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzg2t\" (UniqueName: \"kubernetes.io/projected/0773a31e-b378-4c6a-9f35-7ba46f2e0dc7-kube-api-access-mzg2t\") pod \"payload-processing-e2e-praxis-b-f7f61d-6dcb8fbbb5-n2wb4\" (UID: \"0773a31e-b378-4c6a-9f35-7ba46f2e0dc7\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-f7f61d-6dcb8fbbb5-n2wb4" Nov 03 02:03:59.336656 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:59.336212 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-f7f61d-6dcb8fbbb5-n2wb4" Nov 03 02:03:59.595415 ip-10-0-136-16 kubenswrapper[5437]: W1103 02:03:59.595381 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11b45fe9_41b7_445c_b886_34b15a5e0d8a.slice/crio-022ab0168c30aee2efd9e71f58c43cea0c206b870367d6a5c137fd34d9dbf63b WatchSource:0}: Error finding container 022ab0168c30aee2efd9e71f58c43cea0c206b870367d6a5c137fd34d9dbf63b: Status 404 returned error can't find the container with id 022ab0168c30aee2efd9e71f58c43cea0c206b870367d6a5c137fd34d9dbf63b Nov 03 02:03:59.602550 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:59.599231 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-f7f61d-7cf44f54d7-xg8mf"] Nov 03 02:03:59.631173 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:59.631136 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f7f61d-7cf44f54d7-xg8mf" event={"ID":"11b45fe9-41b7-445c-b886-34b15a5e0d8a","Type":"ContainerStarted","Data":"022ab0168c30aee2efd9e71f58c43cea0c206b870367d6a5c137fd34d9dbf63b"} Nov 03 02:03:59.736932 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:03:59.736894 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-f7f61d-6dcb8fbbb5-n2wb4"] Nov 03 02:04:00.639105 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:00.639057 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-f7f61d-6dcb8fbbb5-n2wb4" event={"ID":"0773a31e-b378-4c6a-9f35-7ba46f2e0dc7","Type":"ContainerStarted","Data":"70acd089881998a1041dca893d73bab0ab3b441d5e59e5243d844e8feb4f6a21"} Nov 03 02:04:00.639655 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:00.639127 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-f7f61d-6dcb8fbbb5-n2wb4" event={"ID":"0773a31e-b378-4c6a-9f35-7ba46f2e0dc7","Type":"ContainerStarted","Data":"2c87fa2c18b9ce2003719f8ed9877fa9d11fcff4659a719570b8c5009cb3f9ce"} Nov 03 02:04:00.639655 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:00.639270 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-f7f61d-6dcb8fbbb5-n2wb4" Nov 03 02:04:00.640938 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:00.640916 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f7f61d-7cf44f54d7-xg8mf" event={"ID":"11b45fe9-41b7-445c-b886-34b15a5e0d8a","Type":"ContainerStarted","Data":"a0ba4c7378aa60988f17a7f9a0275822a563103ed4d882e240532e43221f28e1"} Nov 03 02:04:00.641049 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:00.641000 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f7f61d-7cf44f54d7-xg8mf" Nov 03 02:04:00.759698 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:00.759645 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-f7f61d-6dcb8fbbb5-n2wb4" podStartSLOduration=2.7596308819999997 podStartE2EDuration="2.759630882s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:03:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:04:00.757782119 +0000 UTC m=+2722.787264424" watchObservedRunningTime="2026-11-03 02:04:00.759630882 +0000 UTC m=+2722.789113176" Nov 03 02:04:00.879173 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:00.879113 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f7f61d-7cf44f54d7-xg8mf" podStartSLOduration=2.87909852 podStartE2EDuration="2.87909852s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:03:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:04:00.876256028 +0000 UTC m=+2722.905738336" watchObservedRunningTime="2026-11-03 02:04:00.87909852 +0000 UTC m=+2722.908580814" Nov 03 02:04:03.113501 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:03.113446 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:04:03.314386 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:03.314327 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:04:03.556915 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:03.538067 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:04:06.855721 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:06.855678 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:04:07.518346 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:07.518303 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:04:07.617913 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:07.617881 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-059b23-74cdd47845-8jxpp" Nov 03 02:04:07.741978 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:07.741919 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:04:08.194890 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:08.194852 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:04:11.606567 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:11.606525 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:04:11.649501 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:11.649467 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-f7f61d-6dcb8fbbb5-n2wb4" Nov 03 02:04:11.649830 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:11.649813 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f7f61d-7cf44f54d7-xg8mf" Nov 03 02:04:11.937468 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:11.937339 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:04:12.109685 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:12.109652 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-98a394-openshift-default-7dff487c9f-rwl2t"] Nov 03 02:04:12.113604 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:12.113574 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-98a394-openshift-default-7dff487c9f-rwl2t" Nov 03 02:04:12.173263 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:12.173225 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:04:12.182045 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:12.182011 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-98a394-openshift-default-dockercfg-k4njw\"" Nov 03 02:04:12.184235 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:12.184200 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-98a394-openshift-default-7dff487c9f-rwl2t"] Nov 03 02:04:12.270188 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:12.270082 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80edf8e3-195e-4ccf-b826-27adbf5504ef-istio-podinfo\") pod \"e2e-shared-a-w2-98a394-openshift-default-7dff487c9f-rwl2t\" (UID: \"80edf8e3-195e-4ccf-b826-27adbf5504ef\") " pod="openshift-ingress/e2e-shared-a-w2-98a394-openshift-default-7dff487c9f-rwl2t" Nov 03 02:04:12.270483 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:12.270458 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80edf8e3-195e-4ccf-b826-27adbf5504ef-istio-envoy\") pod \"e2e-shared-a-w2-98a394-openshift-default-7dff487c9f-rwl2t\" (UID: \"80edf8e3-195e-4ccf-b826-27adbf5504ef\") " pod="openshift-ingress/e2e-shared-a-w2-98a394-openshift-default-7dff487c9f-rwl2t" Nov 03 02:04:12.270592 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:12.270495 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80edf8e3-195e-4ccf-b826-27adbf5504ef-istiod-ca-cert\") pod \"e2e-shared-a-w2-98a394-openshift-default-7dff487c9f-rwl2t\" (UID: \"80edf8e3-195e-4ccf-b826-27adbf5504ef\") " pod="openshift-ingress/e2e-shared-a-w2-98a394-openshift-default-7dff487c9f-rwl2t" Nov 03 02:04:12.270592 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:12.270522 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80edf8e3-195e-4ccf-b826-27adbf5504ef-istio-data\") pod \"e2e-shared-a-w2-98a394-openshift-default-7dff487c9f-rwl2t\" (UID: \"80edf8e3-195e-4ccf-b826-27adbf5504ef\") " pod="openshift-ingress/e2e-shared-a-w2-98a394-openshift-default-7dff487c9f-rwl2t" Nov 03 02:04:12.270592 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:12.270582 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80edf8e3-195e-4ccf-b826-27adbf5504ef-istio-token\") pod \"e2e-shared-a-w2-98a394-openshift-default-7dff487c9f-rwl2t\" (UID: \"80edf8e3-195e-4ccf-b826-27adbf5504ef\") " pod="openshift-ingress/e2e-shared-a-w2-98a394-openshift-default-7dff487c9f-rwl2t" Nov 03 02:04:12.270760 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:12.270730 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80edf8e3-195e-4ccf-b826-27adbf5504ef-credential-socket\") pod \"e2e-shared-a-w2-98a394-openshift-default-7dff487c9f-rwl2t\" (UID: \"80edf8e3-195e-4ccf-b826-27adbf5504ef\") " pod="openshift-ingress/e2e-shared-a-w2-98a394-openshift-default-7dff487c9f-rwl2t" Nov 03 02:04:12.270816 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:12.270805 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80edf8e3-195e-4ccf-b826-27adbf5504ef-workload-certs\") pod \"e2e-shared-a-w2-98a394-openshift-default-7dff487c9f-rwl2t\" (UID: \"80edf8e3-195e-4ccf-b826-27adbf5504ef\") " pod="openshift-ingress/e2e-shared-a-w2-98a394-openshift-default-7dff487c9f-rwl2t" Nov 03 02:04:12.270948 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:12.270928 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dl6f7\" (UniqueName: \"kubernetes.io/projected/80edf8e3-195e-4ccf-b826-27adbf5504ef-kube-api-access-dl6f7\") pod \"e2e-shared-a-w2-98a394-openshift-default-7dff487c9f-rwl2t\" (UID: \"80edf8e3-195e-4ccf-b826-27adbf5504ef\") " pod="openshift-ingress/e2e-shared-a-w2-98a394-openshift-default-7dff487c9f-rwl2t" Nov 03 02:04:12.271344 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:12.271317 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80edf8e3-195e-4ccf-b826-27adbf5504ef-workload-socket\") pod \"e2e-shared-a-w2-98a394-openshift-default-7dff487c9f-rwl2t\" (UID: \"80edf8e3-195e-4ccf-b826-27adbf5504ef\") " pod="openshift-ingress/e2e-shared-a-w2-98a394-openshift-default-7dff487c9f-rwl2t" Nov 03 02:04:12.373063 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:12.373016 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80edf8e3-195e-4ccf-b826-27adbf5504ef-workload-certs\") pod \"e2e-shared-a-w2-98a394-openshift-default-7dff487c9f-rwl2t\" (UID: \"80edf8e3-195e-4ccf-b826-27adbf5504ef\") " pod="openshift-ingress/e2e-shared-a-w2-98a394-openshift-default-7dff487c9f-rwl2t" Nov 03 02:04:12.373254 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:12.373157 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dl6f7\" (UniqueName: \"kubernetes.io/projected/80edf8e3-195e-4ccf-b826-27adbf5504ef-kube-api-access-dl6f7\") pod \"e2e-shared-a-w2-98a394-openshift-default-7dff487c9f-rwl2t\" (UID: \"80edf8e3-195e-4ccf-b826-27adbf5504ef\") " pod="openshift-ingress/e2e-shared-a-w2-98a394-openshift-default-7dff487c9f-rwl2t" Nov 03 02:04:12.373254 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:12.373198 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80edf8e3-195e-4ccf-b826-27adbf5504ef-workload-socket\") pod \"e2e-shared-a-w2-98a394-openshift-default-7dff487c9f-rwl2t\" (UID: \"80edf8e3-195e-4ccf-b826-27adbf5504ef\") " pod="openshift-ingress/e2e-shared-a-w2-98a394-openshift-default-7dff487c9f-rwl2t" Nov 03 02:04:12.373254 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:12.373247 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80edf8e3-195e-4ccf-b826-27adbf5504ef-istio-podinfo\") pod \"e2e-shared-a-w2-98a394-openshift-default-7dff487c9f-rwl2t\" (UID: \"80edf8e3-195e-4ccf-b826-27adbf5504ef\") " pod="openshift-ingress/e2e-shared-a-w2-98a394-openshift-default-7dff487c9f-rwl2t" Nov 03 02:04:12.373433 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:12.373276 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80edf8e3-195e-4ccf-b826-27adbf5504ef-istio-envoy\") pod \"e2e-shared-a-w2-98a394-openshift-default-7dff487c9f-rwl2t\" (UID: \"80edf8e3-195e-4ccf-b826-27adbf5504ef\") " pod="openshift-ingress/e2e-shared-a-w2-98a394-openshift-default-7dff487c9f-rwl2t" Nov 03 02:04:12.373433 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:12.373300 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80edf8e3-195e-4ccf-b826-27adbf5504ef-istiod-ca-cert\") pod \"e2e-shared-a-w2-98a394-openshift-default-7dff487c9f-rwl2t\" (UID: \"80edf8e3-195e-4ccf-b826-27adbf5504ef\") " pod="openshift-ingress/e2e-shared-a-w2-98a394-openshift-default-7dff487c9f-rwl2t" Nov 03 02:04:12.373433 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:12.373327 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80edf8e3-195e-4ccf-b826-27adbf5504ef-istio-data\") pod \"e2e-shared-a-w2-98a394-openshift-default-7dff487c9f-rwl2t\" (UID: \"80edf8e3-195e-4ccf-b826-27adbf5504ef\") " pod="openshift-ingress/e2e-shared-a-w2-98a394-openshift-default-7dff487c9f-rwl2t" Nov 03 02:04:12.373433 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:12.373418 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80edf8e3-195e-4ccf-b826-27adbf5504ef-istio-token\") pod \"e2e-shared-a-w2-98a394-openshift-default-7dff487c9f-rwl2t\" (UID: \"80edf8e3-195e-4ccf-b826-27adbf5504ef\") " pod="openshift-ingress/e2e-shared-a-w2-98a394-openshift-default-7dff487c9f-rwl2t" Nov 03 02:04:12.373644 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:12.373482 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80edf8e3-195e-4ccf-b826-27adbf5504ef-workload-certs\") pod \"e2e-shared-a-w2-98a394-openshift-default-7dff487c9f-rwl2t\" (UID: \"80edf8e3-195e-4ccf-b826-27adbf5504ef\") " pod="openshift-ingress/e2e-shared-a-w2-98a394-openshift-default-7dff487c9f-rwl2t" Nov 03 02:04:12.373644 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:12.373567 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80edf8e3-195e-4ccf-b826-27adbf5504ef-credential-socket\") pod \"e2e-shared-a-w2-98a394-openshift-default-7dff487c9f-rwl2t\" (UID: \"80edf8e3-195e-4ccf-b826-27adbf5504ef\") " pod="openshift-ingress/e2e-shared-a-w2-98a394-openshift-default-7dff487c9f-rwl2t" Nov 03 02:04:12.373740 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:12.373658 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80edf8e3-195e-4ccf-b826-27adbf5504ef-workload-socket\") pod \"e2e-shared-a-w2-98a394-openshift-default-7dff487c9f-rwl2t\" (UID: \"80edf8e3-195e-4ccf-b826-27adbf5504ef\") " pod="openshift-ingress/e2e-shared-a-w2-98a394-openshift-default-7dff487c9f-rwl2t" Nov 03 02:04:12.374120 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:12.374072 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80edf8e3-195e-4ccf-b826-27adbf5504ef-istio-data\") pod \"e2e-shared-a-w2-98a394-openshift-default-7dff487c9f-rwl2t\" (UID: \"80edf8e3-195e-4ccf-b826-27adbf5504ef\") " pod="openshift-ingress/e2e-shared-a-w2-98a394-openshift-default-7dff487c9f-rwl2t" Nov 03 02:04:12.374120 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:12.374084 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80edf8e3-195e-4ccf-b826-27adbf5504ef-istiod-ca-cert\") pod \"e2e-shared-a-w2-98a394-openshift-default-7dff487c9f-rwl2t\" (UID: \"80edf8e3-195e-4ccf-b826-27adbf5504ef\") " pod="openshift-ingress/e2e-shared-a-w2-98a394-openshift-default-7dff487c9f-rwl2t" Nov 03 02:04:12.374313 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:12.374150 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80edf8e3-195e-4ccf-b826-27adbf5504ef-credential-socket\") pod \"e2e-shared-a-w2-98a394-openshift-default-7dff487c9f-rwl2t\" (UID: \"80edf8e3-195e-4ccf-b826-27adbf5504ef\") " pod="openshift-ingress/e2e-shared-a-w2-98a394-openshift-default-7dff487c9f-rwl2t" Nov 03 02:04:12.375867 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:12.375838 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80edf8e3-195e-4ccf-b826-27adbf5504ef-istio-envoy\") pod \"e2e-shared-a-w2-98a394-openshift-default-7dff487c9f-rwl2t\" (UID: \"80edf8e3-195e-4ccf-b826-27adbf5504ef\") " pod="openshift-ingress/e2e-shared-a-w2-98a394-openshift-default-7dff487c9f-rwl2t" Nov 03 02:04:12.376081 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:12.376065 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80edf8e3-195e-4ccf-b826-27adbf5504ef-istio-podinfo\") pod \"e2e-shared-a-w2-98a394-openshift-default-7dff487c9f-rwl2t\" (UID: \"80edf8e3-195e-4ccf-b826-27adbf5504ef\") " pod="openshift-ingress/e2e-shared-a-w2-98a394-openshift-default-7dff487c9f-rwl2t" Nov 03 02:04:12.383284 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:12.383240 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:04:12.436170 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:12.436131 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80edf8e3-195e-4ccf-b826-27adbf5504ef-istio-token\") pod \"e2e-shared-a-w2-98a394-openshift-default-7dff487c9f-rwl2t\" (UID: \"80edf8e3-195e-4ccf-b826-27adbf5504ef\") " pod="openshift-ingress/e2e-shared-a-w2-98a394-openshift-default-7dff487c9f-rwl2t" Nov 03 02:04:12.436347 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:12.436211 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dl6f7\" (UniqueName: \"kubernetes.io/projected/80edf8e3-195e-4ccf-b826-27adbf5504ef-kube-api-access-dl6f7\") pod \"e2e-shared-a-w2-98a394-openshift-default-7dff487c9f-rwl2t\" (UID: \"80edf8e3-195e-4ccf-b826-27adbf5504ef\") " pod="openshift-ingress/e2e-shared-a-w2-98a394-openshift-default-7dff487c9f-rwl2t" Nov 03 02:04:12.649822 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:12.649783 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:04:12.729350 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:12.729304 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-98a394-openshift-default-7dff487c9f-rwl2t" Nov 03 02:04:13.154909 ip-10-0-136-16 kubenswrapper[5437]: W1103 02:04:13.154877 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80edf8e3_195e_4ccf_b826_27adbf5504ef.slice/crio-e3c9554a9ca4a5c50feb27dc606efdab67bcce385a87a42ab0c19c75ef150639 WatchSource:0}: Error finding container e3c9554a9ca4a5c50feb27dc606efdab67bcce385a87a42ab0c19c75ef150639: Status 404 returned error can't find the container with id e3c9554a9ca4a5c50feb27dc606efdab67bcce385a87a42ab0c19c75ef150639 Nov 03 02:04:13.157813 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:13.157779 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 02:04:13.157907 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:13.157873 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 02:04:13.157907 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:13.157903 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 02:04:13.157993 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:13.157975 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-98a394-openshift-default-7dff487c9f-rwl2t"] Nov 03 02:04:13.199181 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:13.199148 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:04:13.728646 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:13.728603 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-98a394-openshift-default-7dff487c9f-rwl2t" event={"ID":"80edf8e3-195e-4ccf-b826-27adbf5504ef","Type":"ContainerStarted","Data":"29ac49292258890b95a12c62bf5b229da77fa6281b14f9711a199a784f3f8876"} Nov 03 02:04:13.729056 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:13.728663 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-98a394-openshift-default-7dff487c9f-rwl2t" event={"ID":"80edf8e3-195e-4ccf-b826-27adbf5504ef","Type":"ContainerStarted","Data":"e3c9554a9ca4a5c50feb27dc606efdab67bcce385a87a42ab0c19c75ef150639"} Nov 03 02:04:13.729857 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:13.729837 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-98a394-openshift-default-7dff487c9f-rwl2t" Nov 03 02:04:13.852402 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:13.852327 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-98a394-openshift-default-7dff487c9f-rwl2t" podStartSLOduration=1.852312456 podStartE2EDuration="1.852312456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:04:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:04:13.851954251 +0000 UTC m=+2735.881436548" watchObservedRunningTime="2026-11-03 02:04:13.852312456 +0000 UTC m=+2735.881794750" Nov 03 02:04:14.035857 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:04:14.035770 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 02:04:14.736794 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:14.736760 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-98a394-openshift-default-7dff487c9f-rwl2t" Nov 03 02:04:15.045144 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:15.045056 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:04:15.250130 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:15.250092 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:04:15.397411 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:15.397337 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c34879-6ff9ff78b7-sk2j2"] Nov 03 02:04:15.402115 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:15.402092 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c34879-6ff9ff78b7-sk2j2" Nov 03 02:04:15.415170 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:15.415131 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5vfz\" (UniqueName: \"kubernetes.io/projected/c3d40c05-1fca-4cad-873c-cc68bc420627-kube-api-access-p5vfz\") pod \"payload-pre-processing-e2e-shared-a-w4-c34879-6ff9ff78b7-sk2j2\" (UID: \"c3d40c05-1fca-4cad-873c-cc68bc420627\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c34879-6ff9ff78b7-sk2j2" Nov 03 02:04:15.415329 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:15.415206 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3d40c05-1fca-4cad-873c-cc68bc420627-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-c34879-6ff9ff78b7-sk2j2\" (UID: \"c3d40c05-1fca-4cad-873c-cc68bc420627\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c34879-6ff9ff78b7-sk2j2" Nov 03 02:04:15.455308 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:15.455269 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c34879-6ff9ff78b7-sk2j2"] Nov 03 02:04:15.456803 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:15.456771 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-c34879\"" Nov 03 02:04:15.457552 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:15.457528 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-c34879-dockercfg-2vpp2\"" Nov 03 02:04:15.516377 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:15.516317 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5vfz\" (UniqueName: \"kubernetes.io/projected/c3d40c05-1fca-4cad-873c-cc68bc420627-kube-api-access-p5vfz\") pod \"payload-pre-processing-e2e-shared-a-w4-c34879-6ff9ff78b7-sk2j2\" (UID: \"c3d40c05-1fca-4cad-873c-cc68bc420627\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c34879-6ff9ff78b7-sk2j2" Nov 03 02:04:15.516603 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:15.516523 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3d40c05-1fca-4cad-873c-cc68bc420627-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-c34879-6ff9ff78b7-sk2j2\" (UID: \"c3d40c05-1fca-4cad-873c-cc68bc420627\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c34879-6ff9ff78b7-sk2j2" Nov 03 02:04:15.518323 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:15.518292 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3d40c05-1fca-4cad-873c-cc68bc420627-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-c34879-6ff9ff78b7-sk2j2\" (UID: \"c3d40c05-1fca-4cad-873c-cc68bc420627\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c34879-6ff9ff78b7-sk2j2" Nov 03 02:04:15.558172 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:15.558135 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-c34879-8655cb9dd9-gl2ss"] Nov 03 02:04:15.562582 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:15.562557 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-c34879-8655cb9dd9-gl2ss" Nov 03 02:04:15.579732 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:15.579696 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5vfz\" (UniqueName: \"kubernetes.io/projected/c3d40c05-1fca-4cad-873c-cc68bc420627-kube-api-access-p5vfz\") pod \"payload-pre-processing-e2e-shared-a-w4-c34879-6ff9ff78b7-sk2j2\" (UID: \"c3d40c05-1fca-4cad-873c-cc68bc420627\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c34879-6ff9ff78b7-sk2j2" Nov 03 02:04:15.617557 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:15.617516 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-c34879-8655cb9dd9-gl2ss"] Nov 03 02:04:15.617792 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:15.617706 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xr72b\" (UniqueName: \"kubernetes.io/projected/4cd50b12-718f-4a10-a4f5-16a17fc90342-kube-api-access-xr72b\") pod \"payload-processing-e2e-shared-a-w4-c34879-8655cb9dd9-gl2ss\" (UID: \"4cd50b12-718f-4a10-a4f5-16a17fc90342\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-c34879-8655cb9dd9-gl2ss" Nov 03 02:04:15.617867 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:15.617787 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cd50b12-718f-4a10-a4f5-16a17fc90342-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-c34879-8655cb9dd9-gl2ss\" (UID: \"4cd50b12-718f-4a10-a4f5-16a17fc90342\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-c34879-8655cb9dd9-gl2ss" Nov 03 02:04:15.718435 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:15.718330 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c34879-6ff9ff78b7-sk2j2" Nov 03 02:04:15.718864 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:15.718836 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xr72b\" (UniqueName: \"kubernetes.io/projected/4cd50b12-718f-4a10-a4f5-16a17fc90342-kube-api-access-xr72b\") pod \"payload-processing-e2e-shared-a-w4-c34879-8655cb9dd9-gl2ss\" (UID: \"4cd50b12-718f-4a10-a4f5-16a17fc90342\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-c34879-8655cb9dd9-gl2ss" Nov 03 02:04:15.718939 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:15.718928 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cd50b12-718f-4a10-a4f5-16a17fc90342-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-c34879-8655cb9dd9-gl2ss\" (UID: \"4cd50b12-718f-4a10-a4f5-16a17fc90342\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-c34879-8655cb9dd9-gl2ss" Nov 03 02:04:15.719630 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:15.719608 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cd50b12-718f-4a10-a4f5-16a17fc90342-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-c34879-8655cb9dd9-gl2ss\" (UID: \"4cd50b12-718f-4a10-a4f5-16a17fc90342\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-c34879-8655cb9dd9-gl2ss" Nov 03 02:04:15.743962 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:15.743929 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-98a394-openshift-default-7dff487c9f-rwl2t" Nov 03 02:04:15.745487 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:15.745463 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-98a394-openshift-default-7dff487c9f-rwl2t" Nov 03 02:04:15.783400 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:15.782566 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xr72b\" (UniqueName: \"kubernetes.io/projected/4cd50b12-718f-4a10-a4f5-16a17fc90342-kube-api-access-xr72b\") pod \"payload-processing-e2e-shared-a-w4-c34879-8655cb9dd9-gl2ss\" (UID: \"4cd50b12-718f-4a10-a4f5-16a17fc90342\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-c34879-8655cb9dd9-gl2ss" Nov 03 02:04:15.879319 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:15.879278 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-c34879-8655cb9dd9-gl2ss" Nov 03 02:04:16.093590 ip-10-0-136-16 kubenswrapper[5437]: W1103 02:04:16.093547 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3d40c05_1fca_4cad_873c_cc68bc420627.slice/crio-b2158318de51252c0caa8999e21a350b8704b1010e634dc74f1097ae502324be WatchSource:0}: Error finding container b2158318de51252c0caa8999e21a350b8704b1010e634dc74f1097ae502324be: Status 404 returned error can't find the container with id b2158318de51252c0caa8999e21a350b8704b1010e634dc74f1097ae502324be Nov 03 02:04:16.097699 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:16.097667 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c34879-6ff9ff78b7-sk2j2"] Nov 03 02:04:16.273873 ip-10-0-136-16 kubenswrapper[5437]: W1103 02:04:16.273828 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cd50b12_718f_4a10_a4f5_16a17fc90342.slice/crio-e857ea1f406edc4a04aa17d1d40a7c1a584de3a26944cceac7cefaaf87d49f17 WatchSource:0}: Error finding container e857ea1f406edc4a04aa17d1d40a7c1a584de3a26944cceac7cefaaf87d49f17: Status 404 returned error can't find the container with id e857ea1f406edc4a04aa17d1d40a7c1a584de3a26944cceac7cefaaf87d49f17 Nov 03 02:04:16.277118 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:16.277092 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-c34879-8655cb9dd9-gl2ss"] Nov 03 02:04:16.750212 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:16.750120 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:04:16.752383 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:16.752334 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-c34879-8655cb9dd9-gl2ss" event={"ID":"4cd50b12-718f-4a10-a4f5-16a17fc90342","Type":"ContainerStarted","Data":"d2605c4ed68964e7a345e5ee3478369ebfd139ad35f06157b67f36eefad350ea"} Nov 03 02:04:16.752545 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:16.752412 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-c34879-8655cb9dd9-gl2ss" event={"ID":"4cd50b12-718f-4a10-a4f5-16a17fc90342","Type":"ContainerStarted","Data":"e857ea1f406edc4a04aa17d1d40a7c1a584de3a26944cceac7cefaaf87d49f17"} Nov 03 02:04:16.752597 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:16.752552 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-c34879-8655cb9dd9-gl2ss" Nov 03 02:04:16.754205 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:16.754175 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c34879-6ff9ff78b7-sk2j2" event={"ID":"c3d40c05-1fca-4cad-873c-cc68bc420627","Type":"ContainerStarted","Data":"23b797ac5a126cdd03d1edd848a0f3b7983cce94e062135fd227ebac4ca7ef8e"} Nov 03 02:04:16.754343 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:16.754223 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c34879-6ff9ff78b7-sk2j2" event={"ID":"c3d40c05-1fca-4cad-873c-cc68bc420627","Type":"ContainerStarted","Data":"b2158318de51252c0caa8999e21a350b8704b1010e634dc74f1097ae502324be"} Nov 03 02:04:16.754457 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:16.754441 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c34879-6ff9ff78b7-sk2j2" Nov 03 02:04:16.877338 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:16.877279 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-c34879-8655cb9dd9-gl2ss" podStartSLOduration=1.877264297 podStartE2EDuration="1.877264297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:04:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:04:16.876616292 +0000 UTC m=+2738.906098587" watchObservedRunningTime="2026-11-03 02:04:16.877264297 +0000 UTC m=+2738.906746595" Nov 03 02:04:16.954608 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:16.954570 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:04:16.997725 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:16.997658 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c34879-6ff9ff78b7-sk2j2" podStartSLOduration=1.9976353 podStartE2EDuration="1.9976353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:04:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:04:16.995639617 +0000 UTC m=+2739.025121913" watchObservedRunningTime="2026-11-03 02:04:16.9976353 +0000 UTC m=+2739.027117596" Nov 03 02:04:18.372880 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:18.372642 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:04:22.016929 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:22.016768 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:04:22.272049 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:22.271954 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:04:22.414672 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:22.414638 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-98a394-67d66f5849-ktn8h"] Nov 03 02:04:22.453390 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:22.445816 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-98a394-67d66f5849-ktn8h" Nov 03 02:04:22.504389 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:22.491286 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85af418f-1d3c-44dd-803b-fd73c7e18b0d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-98a394-67d66f5849-ktn8h\" (UID: \"85af418f-1d3c-44dd-803b-fd73c7e18b0d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-98a394-67d66f5849-ktn8h" Nov 03 02:04:22.504389 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:22.491466 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwlz2\" (UniqueName: \"kubernetes.io/projected/85af418f-1d3c-44dd-803b-fd73c7e18b0d-kube-api-access-nwlz2\") pod \"payload-processing-e2e-shared-a-w2-98a394-67d66f5849-ktn8h\" (UID: \"85af418f-1d3c-44dd-803b-fd73c7e18b0d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-98a394-67d66f5849-ktn8h" Nov 03 02:04:22.534857 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:22.526826 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-98a394-67d66f5849-ktn8h"] Nov 03 02:04:22.583640 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:22.583598 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-98a394-dockercfg-686s7\"" Nov 03 02:04:22.585745 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:22.585135 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-98a394\"" Nov 03 02:04:22.600166 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:22.600129 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85af418f-1d3c-44dd-803b-fd73c7e18b0d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-98a394-67d66f5849-ktn8h\" (UID: \"85af418f-1d3c-44dd-803b-fd73c7e18b0d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-98a394-67d66f5849-ktn8h" Nov 03 02:04:22.600801 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:22.600777 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwlz2\" (UniqueName: \"kubernetes.io/projected/85af418f-1d3c-44dd-803b-fd73c7e18b0d-kube-api-access-nwlz2\") pod \"payload-processing-e2e-shared-a-w2-98a394-67d66f5849-ktn8h\" (UID: \"85af418f-1d3c-44dd-803b-fd73c7e18b0d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-98a394-67d66f5849-ktn8h" Nov 03 02:04:22.601339 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:22.601314 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85af418f-1d3c-44dd-803b-fd73c7e18b0d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-98a394-67d66f5849-ktn8h\" (UID: \"85af418f-1d3c-44dd-803b-fd73c7e18b0d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-98a394-67d66f5849-ktn8h" Nov 03 02:04:22.666126 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:22.666076 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwlz2\" (UniqueName: \"kubernetes.io/projected/85af418f-1d3c-44dd-803b-fd73c7e18b0d-kube-api-access-nwlz2\") pod \"payload-processing-e2e-shared-a-w2-98a394-67d66f5849-ktn8h\" (UID: \"85af418f-1d3c-44dd-803b-fd73c7e18b0d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-98a394-67d66f5849-ktn8h" Nov 03 02:04:22.910219 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:22.910175 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-98a394-67d66f5849-ktn8h" Nov 03 02:04:23.288176 ip-10-0-136-16 kubenswrapper[5437]: W1103 02:04:23.288126 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85af418f_1d3c_44dd_803b_fd73c7e18b0d.slice/crio-a2852e30eacd03f073bea0ec3087d1968d076c8541c5b3c2f357adbbc3b857cd WatchSource:0}: Error finding container a2852e30eacd03f073bea0ec3087d1968d076c8541c5b3c2f357adbbc3b857cd: Status 404 returned error can't find the container with id a2852e30eacd03f073bea0ec3087d1968d076c8541c5b3c2f357adbbc3b857cd Nov 03 02:04:23.291287 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:23.291254 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-98a394-67d66f5849-ktn8h"] Nov 03 02:04:23.390966 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:23.390928 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-kzj4s"] Nov 03 02:04:23.395018 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:23.394992 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-kzj4s" Nov 03 02:04:23.512506 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:23.512466 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ced2a84a-85cd-4435-bbca-33ee4ae19496-istio-podinfo\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-kzj4s\" (UID: \"ced2a84a-85cd-4435-bbca-33ee4ae19496\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-kzj4s" Nov 03 02:04:23.512696 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:23.512528 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ced2a84a-85cd-4435-bbca-33ee4ae19496-istiod-ca-cert\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-kzj4s\" (UID: \"ced2a84a-85cd-4435-bbca-33ee4ae19496\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-kzj4s" Nov 03 02:04:23.512696 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:23.512681 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ced2a84a-85cd-4435-bbca-33ee4ae19496-istio-envoy\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-kzj4s\" (UID: \"ced2a84a-85cd-4435-bbca-33ee4ae19496\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-kzj4s" Nov 03 02:04:23.512820 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:23.512724 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ced2a84a-85cd-4435-bbca-33ee4ae19496-istio-data\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-kzj4s\" (UID: \"ced2a84a-85cd-4435-bbca-33ee4ae19496\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-kzj4s" Nov 03 02:04:23.512871 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:23.512856 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ced2a84a-85cd-4435-bbca-33ee4ae19496-credential-socket\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-kzj4s\" (UID: \"ced2a84a-85cd-4435-bbca-33ee4ae19496\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-kzj4s" Nov 03 02:04:23.512956 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:23.512941 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ced2a84a-85cd-4435-bbca-33ee4ae19496-workload-certs\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-kzj4s\" (UID: \"ced2a84a-85cd-4435-bbca-33ee4ae19496\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-kzj4s" Nov 03 02:04:23.513009 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:23.512976 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ced2a84a-85cd-4435-bbca-33ee4ae19496-workload-socket\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-kzj4s\" (UID: \"ced2a84a-85cd-4435-bbca-33ee4ae19496\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-kzj4s" Nov 03 02:04:23.513046 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:23.513024 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2n89j\" (UniqueName: \"kubernetes.io/projected/ced2a84a-85cd-4435-bbca-33ee4ae19496-kube-api-access-2n89j\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-kzj4s\" (UID: \"ced2a84a-85cd-4435-bbca-33ee4ae19496\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-kzj4s" Nov 03 02:04:23.513078 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:23.513069 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ced2a84a-85cd-4435-bbca-33ee4ae19496-istio-token\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-kzj4s\" (UID: \"ced2a84a-85cd-4435-bbca-33ee4ae19496\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-kzj4s" Nov 03 02:04:23.602385 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:23.602335 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-kzj4s"] Nov 03 02:04:23.614246 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:23.614202 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ced2a84a-85cd-4435-bbca-33ee4ae19496-workload-certs\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-kzj4s\" (UID: \"ced2a84a-85cd-4435-bbca-33ee4ae19496\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-kzj4s" Nov 03 02:04:23.614482 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:23.614284 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ced2a84a-85cd-4435-bbca-33ee4ae19496-workload-socket\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-kzj4s\" (UID: \"ced2a84a-85cd-4435-bbca-33ee4ae19496\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-kzj4s" Nov 03 02:04:23.614482 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:23.614328 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2n89j\" (UniqueName: \"kubernetes.io/projected/ced2a84a-85cd-4435-bbca-33ee4ae19496-kube-api-access-2n89j\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-kzj4s\" (UID: \"ced2a84a-85cd-4435-bbca-33ee4ae19496\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-kzj4s" Nov 03 02:04:23.614482 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:23.614417 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ced2a84a-85cd-4435-bbca-33ee4ae19496-istio-token\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-kzj4s\" (UID: \"ced2a84a-85cd-4435-bbca-33ee4ae19496\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-kzj4s" Nov 03 02:04:23.614656 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:23.614499 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ced2a84a-85cd-4435-bbca-33ee4ae19496-istio-podinfo\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-kzj4s\" (UID: \"ced2a84a-85cd-4435-bbca-33ee4ae19496\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-kzj4s" Nov 03 02:04:23.614656 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:23.614538 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ced2a84a-85cd-4435-bbca-33ee4ae19496-istiod-ca-cert\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-kzj4s\" (UID: \"ced2a84a-85cd-4435-bbca-33ee4ae19496\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-kzj4s" Nov 03 02:04:23.614656 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:23.614599 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ced2a84a-85cd-4435-bbca-33ee4ae19496-istio-envoy\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-kzj4s\" (UID: \"ced2a84a-85cd-4435-bbca-33ee4ae19496\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-kzj4s" Nov 03 02:04:23.614656 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:23.614630 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ced2a84a-85cd-4435-bbca-33ee4ae19496-istio-data\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-kzj4s\" (UID: \"ced2a84a-85cd-4435-bbca-33ee4ae19496\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-kzj4s" Nov 03 02:04:23.614865 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:23.614696 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ced2a84a-85cd-4435-bbca-33ee4ae19496-workload-certs\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-kzj4s\" (UID: \"ced2a84a-85cd-4435-bbca-33ee4ae19496\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-kzj4s" Nov 03 02:04:23.614865 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:23.614765 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ced2a84a-85cd-4435-bbca-33ee4ae19496-credential-socket\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-kzj4s\" (UID: \"ced2a84a-85cd-4435-bbca-33ee4ae19496\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-kzj4s" Nov 03 02:04:23.614865 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:23.614769 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ced2a84a-85cd-4435-bbca-33ee4ae19496-workload-socket\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-kzj4s\" (UID: \"ced2a84a-85cd-4435-bbca-33ee4ae19496\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-kzj4s" Nov 03 02:04:23.615065 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:23.615041 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ced2a84a-85cd-4435-bbca-33ee4ae19496-istio-data\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-kzj4s\" (UID: \"ced2a84a-85cd-4435-bbca-33ee4ae19496\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-kzj4s" Nov 03 02:04:23.615131 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:23.615096 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ced2a84a-85cd-4435-bbca-33ee4ae19496-credential-socket\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-kzj4s\" (UID: \"ced2a84a-85cd-4435-bbca-33ee4ae19496\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-kzj4s" Nov 03 02:04:23.615302 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:23.615275 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ced2a84a-85cd-4435-bbca-33ee4ae19496-istiod-ca-cert\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-kzj4s\" (UID: \"ced2a84a-85cd-4435-bbca-33ee4ae19496\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-kzj4s" Nov 03 02:04:23.617580 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:23.617550 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ced2a84a-85cd-4435-bbca-33ee4ae19496-istio-envoy\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-kzj4s\" (UID: \"ced2a84a-85cd-4435-bbca-33ee4ae19496\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-kzj4s" Nov 03 02:04:23.617736 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:23.617715 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ced2a84a-85cd-4435-bbca-33ee4ae19496-istio-podinfo\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-kzj4s\" (UID: \"ced2a84a-85cd-4435-bbca-33ee4ae19496\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-kzj4s" Nov 03 02:04:23.678662 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:23.678597 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ced2a84a-85cd-4435-bbca-33ee4ae19496-istio-token\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-kzj4s\" (UID: \"ced2a84a-85cd-4435-bbca-33ee4ae19496\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-kzj4s" Nov 03 02:04:23.678944 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:23.678761 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2n89j\" (UniqueName: \"kubernetes.io/projected/ced2a84a-85cd-4435-bbca-33ee4ae19496-kube-api-access-2n89j\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-kzj4s\" (UID: \"ced2a84a-85cd-4435-bbca-33ee4ae19496\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-kzj4s" Nov 03 02:04:23.686901 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:23.686855 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:04:23.710124 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:23.710079 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-kzj4s" Nov 03 02:04:23.820311 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:23.820273 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-98a394-67d66f5849-ktn8h" event={"ID":"85af418f-1d3c-44dd-803b-fd73c7e18b0d","Type":"ContainerStarted","Data":"3230d5692cbf8f6be880387103819069e94c4908bf678210dccb91ba64341d97"} Nov 03 02:04:23.820577 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:23.820332 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-98a394-67d66f5849-ktn8h" event={"ID":"85af418f-1d3c-44dd-803b-fd73c7e18b0d","Type":"ContainerStarted","Data":"a2852e30eacd03f073bea0ec3087d1968d076c8541c5b3c2f357adbbc3b857cd"} Nov 03 02:04:23.820577 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:23.820496 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-98a394-67d66f5849-ktn8h" Nov 03 02:04:23.942072 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:23.941963 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-98a394-67d66f5849-ktn8h" podStartSLOduration=1.94194723 podStartE2EDuration="1.94194723s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:04:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:04:23.93985465 +0000 UTC m=+2745.969336945" watchObservedRunningTime="2026-11-03 02:04:23.94194723 +0000 UTC m=+2745.971429525" Nov 03 02:04:24.136205 ip-10-0-136-16 kubenswrapper[5437]: W1103 02:04:24.136166 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podced2a84a_85cd_4435_bbca_33ee4ae19496.slice/crio-02171d783cae11aefcf8460cb2df79a89905d45cc7cdd5b079e0fd23955a4b03 WatchSource:0}: Error finding container 02171d783cae11aefcf8460cb2df79a89905d45cc7cdd5b079e0fd23955a4b03: Status 404 returned error can't find the container with id 02171d783cae11aefcf8460cb2df79a89905d45cc7cdd5b079e0fd23955a4b03 Nov 03 02:04:24.139038 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:24.139002 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 02:04:24.139157 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:24.139076 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 02:04:24.139157 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:24.139106 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 02:04:24.139375 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:24.139342 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-kzj4s"] Nov 03 02:04:24.836473 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:24.836429 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-kzj4s" event={"ID":"ced2a84a-85cd-4435-bbca-33ee4ae19496","Type":"ContainerStarted","Data":"ba41559d0eeaf4d3a5557e8b0700d53f0c491d8b40c1fafeba2241cb14129d1d"} Nov 03 02:04:24.837110 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:24.836523 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-kzj4s" event={"ID":"ced2a84a-85cd-4435-bbca-33ee4ae19496","Type":"ContainerStarted","Data":"02171d783cae11aefcf8460cb2df79a89905d45cc7cdd5b079e0fd23955a4b03"} Nov 03 02:04:25.010546 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:25.010471 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-kzj4s" podStartSLOduration=2.010449221 podStartE2EDuration="2.010449221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:04:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:04:25.007896977 +0000 UTC m=+2747.037379269" watchObservedRunningTime="2026-11-03 02:04:25.010449221 +0000 UTC m=+2747.039931517" Nov 03 02:04:25.711095 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:25.711045 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-kzj4s" Nov 03 02:04:25.713032 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:25.713001 5437 patch_prober.go:29] interesting pod/e2e-worker-w2-9da431-openshift-default-65c799fd77-kzj4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body= Nov 03 02:04:25.713193 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:25.713057 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-kzj4s" podUID="ced2a84a-85cd-4435-bbca-33ee4ae19496" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" Nov 03 02:04:26.711197 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:26.711160 5437 patch_prober.go:29] interesting pod/e2e-worker-w2-9da431-openshift-default-65c799fd77-kzj4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body= Nov 03 02:04:26.711653 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:26.711220 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-kzj4s" podUID="ced2a84a-85cd-4435-bbca-33ee4ae19496" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" Nov 03 02:04:27.711390 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:27.711325 5437 patch_prober.go:29] interesting pod/e2e-worker-w2-9da431-openshift-default-65c799fd77-kzj4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body= Nov 03 02:04:27.711825 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:27.711424 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-kzj4s" podUID="ced2a84a-85cd-4435-bbca-33ee4ae19496" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" Nov 03 02:04:27.772727 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:27.772693 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-c34879-8655cb9dd9-gl2ss" Nov 03 02:04:27.772909 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:27.772779 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c34879-6ff9ff78b7-sk2j2" Nov 03 02:04:28.711287 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:28.711250 5437 patch_prober.go:29] interesting pod/e2e-worker-w2-9da431-openshift-default-65c799fd77-kzj4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body= Nov 03 02:04:28.711524 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:28.711312 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-kzj4s" podUID="ced2a84a-85cd-4435-bbca-33ee4ae19496" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" Nov 03 02:04:28.762861 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:28.762811 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:04:29.096914 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:29.096841 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:04:29.306427 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:29.306385 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:04:29.710507 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:29.710469 5437 patch_prober.go:29] interesting pod/e2e-worker-w2-9da431-openshift-default-65c799fd77-kzj4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body= Nov 03 02:04:29.710705 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:29.710534 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-kzj4s" podUID="ced2a84a-85cd-4435-bbca-33ee4ae19496" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" Nov 03 02:04:30.710521 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:30.710432 5437 patch_prober.go:29] interesting pod/e2e-worker-w2-9da431-openshift-default-65c799fd77-kzj4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body= Nov 03 02:04:30.710521 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:30.710488 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-kzj4s" podUID="ced2a84a-85cd-4435-bbca-33ee4ae19496" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" Nov 03 02:04:31.814704 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:31.814664 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-kzj4s" Nov 03 02:04:31.815553 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:31.815523 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-kzj4s" Nov 03 02:04:31.816416 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:31.816389 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-kzj4s" Nov 03 02:04:33.935440 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:33.935395 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:04:34.195143 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:34.195031 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:04:34.839662 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:34.839626 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-98a394-67d66f5849-ktn8h" Nov 03 02:04:39.088173 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:39.088128 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:04:43.580459 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:43.580425 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a54cab-7b76bc44ff-9g8h8"] Nov 03 02:04:43.584790 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:43.584769 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a54cab-7b76bc44ff-9g8h8" Nov 03 02:04:43.639947 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:43.639898 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a54cab-7b76bc44ff-9g8h8"] Nov 03 02:04:43.683670 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:43.683622 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f073cb08-11df-43ab-bb6a-79331e93ac08-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-a54cab-7b76bc44ff-9g8h8\" (UID: \"f073cb08-11df-43ab-bb6a-79331e93ac08\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a54cab-7b76bc44ff-9g8h8" Nov 03 02:04:43.683859 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:43.683765 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f073cb08-11df-43ab-bb6a-79331e93ac08-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-a54cab-7b76bc44ff-9g8h8\" (UID: \"f073cb08-11df-43ab-bb6a-79331e93ac08\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a54cab-7b76bc44ff-9g8h8" Nov 03 02:04:43.683859 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:43.683840 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f073cb08-11df-43ab-bb6a-79331e93ac08-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-a54cab-7b76bc44ff-9g8h8\" (UID: \"f073cb08-11df-43ab-bb6a-79331e93ac08\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a54cab-7b76bc44ff-9g8h8" Nov 03 02:04:43.683993 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:43.683978 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xngf\" (UniqueName: \"kubernetes.io/projected/f073cb08-11df-43ab-bb6a-79331e93ac08-kube-api-access-2xngf\") pod \"maas-api-e2e-shared-a-w1-a54cab-7b76bc44ff-9g8h8\" (UID: \"f073cb08-11df-43ab-bb6a-79331e93ac08\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a54cab-7b76bc44ff-9g8h8" Nov 03 02:04:43.688501 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:43.688477 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w1-a54cab-serving-cert\"" Nov 03 02:04:43.784632 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:43.784590 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f073cb08-11df-43ab-bb6a-79331e93ac08-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-a54cab-7b76bc44ff-9g8h8\" (UID: \"f073cb08-11df-43ab-bb6a-79331e93ac08\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a54cab-7b76bc44ff-9g8h8" Nov 03 02:04:43.784822 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:43.784715 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xngf\" (UniqueName: \"kubernetes.io/projected/f073cb08-11df-43ab-bb6a-79331e93ac08-kube-api-access-2xngf\") pod \"maas-api-e2e-shared-a-w1-a54cab-7b76bc44ff-9g8h8\" (UID: \"f073cb08-11df-43ab-bb6a-79331e93ac08\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a54cab-7b76bc44ff-9g8h8" Nov 03 02:04:43.784822 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:43.784760 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f073cb08-11df-43ab-bb6a-79331e93ac08-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-a54cab-7b76bc44ff-9g8h8\" (UID: \"f073cb08-11df-43ab-bb6a-79331e93ac08\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a54cab-7b76bc44ff-9g8h8" Nov 03 02:04:43.784932 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:43.784822 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f073cb08-11df-43ab-bb6a-79331e93ac08-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-a54cab-7b76bc44ff-9g8h8\" (UID: \"f073cb08-11df-43ab-bb6a-79331e93ac08\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a54cab-7b76bc44ff-9g8h8" Nov 03 02:04:43.784994 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:04:43.784957 5437 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a54cab-serving-cert: secret "maas-api-e2e-shared-a-w1-a54cab-serving-cert" not found Nov 03 02:04:43.785055 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:04:43.785043 5437 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f073cb08-11df-43ab-bb6a-79331e93ac08-maas-api-tls podName:f073cb08-11df-43ab-bb6a-79331e93ac08 nodeName:}" failed. No retries permitted until 2026-11-03 02:04:44.28501964 +0000 UTC m=+2766.314501916 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f073cb08-11df-43ab-bb6a-79331e93ac08-maas-api-tls") pod "maas-api-e2e-shared-a-w1-a54cab-7b76bc44ff-9g8h8" (UID: "f073cb08-11df-43ab-bb6a-79331e93ac08") : secret "maas-api-e2e-shared-a-w1-a54cab-serving-cert" not found Nov 03 02:04:43.785409 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:43.785358 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f073cb08-11df-43ab-bb6a-79331e93ac08-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-a54cab-7b76bc44ff-9g8h8\" (UID: \"f073cb08-11df-43ab-bb6a-79331e93ac08\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a54cab-7b76bc44ff-9g8h8" Nov 03 02:04:43.787061 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:43.787036 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f073cb08-11df-43ab-bb6a-79331e93ac08-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-a54cab-7b76bc44ff-9g8h8\" (UID: \"f073cb08-11df-43ab-bb6a-79331e93ac08\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a54cab-7b76bc44ff-9g8h8" Nov 03 02:04:43.839137 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:43.839038 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:04:43.850477 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:43.850443 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xngf\" (UniqueName: \"kubernetes.io/projected/f073cb08-11df-43ab-bb6a-79331e93ac08-kube-api-access-2xngf\") pod \"maas-api-e2e-shared-a-w1-a54cab-7b76bc44ff-9g8h8\" (UID: \"f073cb08-11df-43ab-bb6a-79331e93ac08\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a54cab-7b76bc44ff-9g8h8" Nov 03 02:04:44.046793 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:04:44.046748 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 02:04:44.056300 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:44.056259 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:04:44.281847 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:44.281747 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:04:44.292260 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:44.292213 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f073cb08-11df-43ab-bb6a-79331e93ac08-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-a54cab-7b76bc44ff-9g8h8\" (UID: \"f073cb08-11df-43ab-bb6a-79331e93ac08\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a54cab-7b76bc44ff-9g8h8" Nov 03 02:04:44.294796 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:44.294771 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f073cb08-11df-43ab-bb6a-79331e93ac08-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-a54cab-7b76bc44ff-9g8h8\" (UID: \"f073cb08-11df-43ab-bb6a-79331e93ac08\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a54cab-7b76bc44ff-9g8h8" Nov 03 02:04:44.498216 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:44.498176 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a54cab-7b76bc44ff-9g8h8" Nov 03 02:04:44.878754 ip-10-0-136-16 kubenswrapper[5437]: W1103 02:04:44.878716 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf073cb08_11df_43ab_bb6a_79331e93ac08.slice/crio-68111bf0de62a89013c7f311c3541dd070454c9d1236e8cf2bc31462b16c108a WatchSource:0}: Error finding container 68111bf0de62a89013c7f311c3541dd070454c9d1236e8cf2bc31462b16c108a: Status 404 returned error can't find the container with id 68111bf0de62a89013c7f311c3541dd070454c9d1236e8cf2bc31462b16c108a Nov 03 02:04:44.880334 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:44.880311 5437 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 02:04:44.881992 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:44.881969 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a54cab-7b76bc44ff-9g8h8"] Nov 03 02:04:44.960682 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:44.960649 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a54cab-7b76bc44ff-9g8h8" event={"ID":"f073cb08-11df-43ab-bb6a-79331e93ac08","Type":"ContainerStarted","Data":"68111bf0de62a89013c7f311c3541dd070454c9d1236e8cf2bc31462b16c108a"} Nov 03 02:04:46.975303 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:46.975267 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a54cab-7b76bc44ff-9g8h8" event={"ID":"f073cb08-11df-43ab-bb6a-79331e93ac08","Type":"ContainerStarted","Data":"0a5f608f765f2b3c98aad35b4b34d8ec9c37b6bc81b11734c74af5015640c1ba"} Nov 03 02:04:46.975767 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:46.975451 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a54cab-7b76bc44ff-9g8h8" Nov 03 02:04:47.100198 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:47.100138 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a54cab-7b76bc44ff-9g8h8" podStartSLOduration=2.860970087 podStartE2EDuration="4.100122844s" totalImagesPullingTime="1.239152757s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:04:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:04:47.099563986 +0000 UTC m=+2769.129046280" watchObservedRunningTime="2026-11-03 02:04:47.100122844 +0000 UTC m=+2769.129605143" Nov 03 02:04:48.227963 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:48.227918 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:04:48.863511 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:48.863472 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:04:49.200333 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:49.200251 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-92zjz"] Nov 03 02:04:49.204624 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:49.204598 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-92zjz" Nov 03 02:04:49.258973 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:49.258935 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/748c5d2d-505a-4ac4-8563-d445b5ba48de-istio-podinfo\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-92zjz\" (UID: \"748c5d2d-505a-4ac4-8563-d445b5ba48de\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-92zjz" Nov 03 02:04:49.259430 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:49.258999 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/748c5d2d-505a-4ac4-8563-d445b5ba48de-workload-certs\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-92zjz\" (UID: \"748c5d2d-505a-4ac4-8563-d445b5ba48de\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-92zjz" Nov 03 02:04:49.259430 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:49.259108 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/748c5d2d-505a-4ac4-8563-d445b5ba48de-istio-envoy\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-92zjz\" (UID: \"748c5d2d-505a-4ac4-8563-d445b5ba48de\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-92zjz" Nov 03 02:04:49.259430 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:49.259152 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72j2b\" (UniqueName: \"kubernetes.io/projected/748c5d2d-505a-4ac4-8563-d445b5ba48de-kube-api-access-72j2b\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-92zjz\" (UID: \"748c5d2d-505a-4ac4-8563-d445b5ba48de\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-92zjz" Nov 03 02:04:49.259430 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:49.259173 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/748c5d2d-505a-4ac4-8563-d445b5ba48de-istio-token\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-92zjz\" (UID: \"748c5d2d-505a-4ac4-8563-d445b5ba48de\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-92zjz" Nov 03 02:04:49.259430 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:49.259255 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/748c5d2d-505a-4ac4-8563-d445b5ba48de-istio-data\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-92zjz\" (UID: \"748c5d2d-505a-4ac4-8563-d445b5ba48de\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-92zjz" Nov 03 02:04:49.259430 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:49.259331 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/748c5d2d-505a-4ac4-8563-d445b5ba48de-workload-socket\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-92zjz\" (UID: \"748c5d2d-505a-4ac4-8563-d445b5ba48de\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-92zjz" Nov 03 02:04:49.259770 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:49.259453 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/748c5d2d-505a-4ac4-8563-d445b5ba48de-credential-socket\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-92zjz\" (UID: \"748c5d2d-505a-4ac4-8563-d445b5ba48de\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-92zjz" Nov 03 02:04:49.259770 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:49.259488 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/748c5d2d-505a-4ac4-8563-d445b5ba48de-istiod-ca-cert\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-92zjz\" (UID: \"748c5d2d-505a-4ac4-8563-d445b5ba48de\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-92zjz" Nov 03 02:04:49.315097 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:49.315053 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-92zjz"] Nov 03 02:04:49.360905 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:49.360864 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/748c5d2d-505a-4ac4-8563-d445b5ba48de-workload-socket\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-92zjz\" (UID: \"748c5d2d-505a-4ac4-8563-d445b5ba48de\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-92zjz" Nov 03 02:04:49.361108 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:49.360957 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/748c5d2d-505a-4ac4-8563-d445b5ba48de-credential-socket\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-92zjz\" (UID: \"748c5d2d-505a-4ac4-8563-d445b5ba48de\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-92zjz" Nov 03 02:04:49.361108 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:49.360994 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/748c5d2d-505a-4ac4-8563-d445b5ba48de-istiod-ca-cert\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-92zjz\" (UID: \"748c5d2d-505a-4ac4-8563-d445b5ba48de\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-92zjz" Nov 03 02:04:49.361108 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:49.361066 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/748c5d2d-505a-4ac4-8563-d445b5ba48de-istio-podinfo\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-92zjz\" (UID: \"748c5d2d-505a-4ac4-8563-d445b5ba48de\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-92zjz" Nov 03 02:04:49.361294 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:49.361128 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/748c5d2d-505a-4ac4-8563-d445b5ba48de-workload-certs\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-92zjz\" (UID: \"748c5d2d-505a-4ac4-8563-d445b5ba48de\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-92zjz" Nov 03 02:04:49.361294 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:49.361176 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/748c5d2d-505a-4ac4-8563-d445b5ba48de-istio-envoy\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-92zjz\" (UID: \"748c5d2d-505a-4ac4-8563-d445b5ba48de\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-92zjz" Nov 03 02:04:49.361294 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:49.361217 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72j2b\" (UniqueName: \"kubernetes.io/projected/748c5d2d-505a-4ac4-8563-d445b5ba48de-kube-api-access-72j2b\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-92zjz\" (UID: \"748c5d2d-505a-4ac4-8563-d445b5ba48de\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-92zjz" Nov 03 02:04:49.361294 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:49.361246 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/748c5d2d-505a-4ac4-8563-d445b5ba48de-istio-token\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-92zjz\" (UID: \"748c5d2d-505a-4ac4-8563-d445b5ba48de\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-92zjz" Nov 03 02:04:49.361294 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:49.361278 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/748c5d2d-505a-4ac4-8563-d445b5ba48de-istio-data\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-92zjz\" (UID: \"748c5d2d-505a-4ac4-8563-d445b5ba48de\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-92zjz" Nov 03 02:04:49.361611 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:49.361402 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/748c5d2d-505a-4ac4-8563-d445b5ba48de-credential-socket\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-92zjz\" (UID: \"748c5d2d-505a-4ac4-8563-d445b5ba48de\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-92zjz" Nov 03 02:04:49.361611 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:49.361543 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/748c5d2d-505a-4ac4-8563-d445b5ba48de-workload-certs\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-92zjz\" (UID: \"748c5d2d-505a-4ac4-8563-d445b5ba48de\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-92zjz" Nov 03 02:04:49.361713 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:49.361655 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/748c5d2d-505a-4ac4-8563-d445b5ba48de-workload-socket\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-92zjz\" (UID: \"748c5d2d-505a-4ac4-8563-d445b5ba48de\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-92zjz" Nov 03 02:04:49.361749 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:49.361733 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/748c5d2d-505a-4ac4-8563-d445b5ba48de-istio-data\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-92zjz\" (UID: \"748c5d2d-505a-4ac4-8563-d445b5ba48de\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-92zjz" Nov 03 02:04:49.361836 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:49.361809 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/748c5d2d-505a-4ac4-8563-d445b5ba48de-istiod-ca-cert\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-92zjz\" (UID: \"748c5d2d-505a-4ac4-8563-d445b5ba48de\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-92zjz" Nov 03 02:04:49.363598 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:49.363572 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/748c5d2d-505a-4ac4-8563-d445b5ba48de-istio-envoy\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-92zjz\" (UID: \"748c5d2d-505a-4ac4-8563-d445b5ba48de\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-92zjz" Nov 03 02:04:49.363996 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:49.363979 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/748c5d2d-505a-4ac4-8563-d445b5ba48de-istio-podinfo\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-92zjz\" (UID: \"748c5d2d-505a-4ac4-8563-d445b5ba48de\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-92zjz" Nov 03 02:04:49.424070 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:49.424025 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/748c5d2d-505a-4ac4-8563-d445b5ba48de-istio-token\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-92zjz\" (UID: \"748c5d2d-505a-4ac4-8563-d445b5ba48de\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-92zjz" Nov 03 02:04:49.424251 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:49.424164 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72j2b\" (UniqueName: \"kubernetes.io/projected/748c5d2d-505a-4ac4-8563-d445b5ba48de-kube-api-access-72j2b\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-92zjz\" (UID: \"748c5d2d-505a-4ac4-8563-d445b5ba48de\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-92zjz" Nov 03 02:04:49.518620 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:49.518526 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-92zjz" Nov 03 02:04:49.941218 ip-10-0-136-16 kubenswrapper[5437]: W1103 02:04:49.941179 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod748c5d2d_505a_4ac4_8563_d445b5ba48de.slice/crio-c3da2fd167c9b99a14e9de4bdbd9e116578ba991cf8555cb97ac5759bfbfd98f WatchSource:0}: Error finding container c3da2fd167c9b99a14e9de4bdbd9e116578ba991cf8555cb97ac5759bfbfd98f: Status 404 returned error can't find the container with id c3da2fd167c9b99a14e9de4bdbd9e116578ba991cf8555cb97ac5759bfbfd98f Nov 03 02:04:49.944077 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:49.944036 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 02:04:49.944192 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:49.944112 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 02:04:49.944192 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:49.944139 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 02:04:49.945297 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:49.945275 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-92zjz"] Nov 03 02:04:49.998677 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:49.998646 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-92zjz" event={"ID":"748c5d2d-505a-4ac4-8563-d445b5ba48de","Type":"ContainerStarted","Data":"c3da2fd167c9b99a14e9de4bdbd9e116578ba991cf8555cb97ac5759bfbfd98f"} Nov 03 02:04:51.008181 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:51.008136 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-92zjz" event={"ID":"748c5d2d-505a-4ac4-8563-d445b5ba48de","Type":"ContainerStarted","Data":"bb1e63d91d134c2b2812ece736cfcce960f30b8524eedf283f6d09fe493edf2b"} Nov 03 02:04:51.244912 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:51.244850 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-92zjz" podStartSLOduration=2.244832748 podStartE2EDuration="2.244832748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:04:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:04:51.18341809 +0000 UTC m=+2773.212900384" watchObservedRunningTime="2026-11-03 02:04:51.244832748 +0000 UTC m=+2773.274315042" Nov 03 02:04:51.525439 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:51.521687 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-92zjz" Nov 03 02:04:51.542606 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:51.542557 5437 patch_prober.go:29] interesting pod/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-92zjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Nov 03 02:04:51.542818 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:51.542635 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-92zjz" podUID="748c5d2d-505a-4ac4-8563-d445b5ba48de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Nov 03 02:04:51.544013 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:51.543982 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:04:52.529394 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:52.520078 5437 patch_prober.go:29] interesting pod/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-92zjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Nov 03 02:04:52.529394 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:52.520150 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-92zjz" podUID="748c5d2d-505a-4ac4-8563-d445b5ba48de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Nov 03 02:04:52.990951 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:52.990919 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a54cab-7b76bc44ff-9g8h8" Nov 03 02:04:53.539391 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:53.528771 5437 patch_prober.go:29] interesting pod/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-92zjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Nov 03 02:04:53.539391 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:53.528839 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-92zjz" podUID="748c5d2d-505a-4ac4-8563-d445b5ba48de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Nov 03 02:04:53.680583 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:53.680540 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:04:54.538864 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:54.538824 5437 patch_prober.go:29] interesting pod/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-92zjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Nov 03 02:04:54.539135 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:54.539107 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-92zjz" podUID="748c5d2d-505a-4ac4-8563-d445b5ba48de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Nov 03 02:04:55.538389 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:55.538174 5437 patch_prober.go:29] interesting pod/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-92zjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Nov 03 02:04:55.538389 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:55.538236 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-92zjz" podUID="748c5d2d-505a-4ac4-8563-d445b5ba48de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Nov 03 02:04:56.286812 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:56.286758 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:04:56.519635 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:56.519597 5437 patch_prober.go:29] interesting pod/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-92zjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Nov 03 02:04:56.519835 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:56.519657 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-92zjz" podUID="748c5d2d-505a-4ac4-8563-d445b5ba48de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Nov 03 02:04:57.523992 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:57.523956 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-92zjz" Nov 03 02:04:57.524439 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:57.524259 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-92zjz" Nov 03 02:04:57.525127 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:57.525104 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-92zjz" Nov 03 02:04:58.436941 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:04:58.436905 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:05:03.151185 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:05:03.151142 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:05:03.403437 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:05:03.403286 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:05:04.605033 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:05:04.604990 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:05:04.810500 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:05:04.810463 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:05:08.083438 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:05:08.083351 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:05:12.489897 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:05:12.489863 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-98a394-d995767bd-68hkm"] Nov 03 02:05:12.494081 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:05:12.494057 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-98a394-d995767bd-68hkm" Nov 03 02:05:12.550207 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:05:12.550173 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-98a394-serving-cert\"" Nov 03 02:05:12.579582 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:05:12.579540 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgq28\" (UniqueName: \"kubernetes.io/projected/bcc4bb00-2f9e-4f38-a331-f725729873f0-kube-api-access-jgq28\") pod \"maas-api-e2e-shared-a-w2-98a394-d995767bd-68hkm\" (UID: \"bcc4bb00-2f9e-4f38-a331-f725729873f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-98a394-d995767bd-68hkm" Nov 03 02:05:12.579775 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:05:12.579614 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcc4bb00-2f9e-4f38-a331-f725729873f0-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-98a394-d995767bd-68hkm\" (UID: \"bcc4bb00-2f9e-4f38-a331-f725729873f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-98a394-d995767bd-68hkm" Nov 03 02:05:12.579835 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:05:12.579805 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bcc4bb00-2f9e-4f38-a331-f725729873f0-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-98a394-d995767bd-68hkm\" (UID: \"bcc4bb00-2f9e-4f38-a331-f725729873f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-98a394-d995767bd-68hkm" Nov 03 02:05:12.579896 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:05:12.579870 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcc4bb00-2f9e-4f38-a331-f725729873f0-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-98a394-d995767bd-68hkm\" (UID: \"bcc4bb00-2f9e-4f38-a331-f725729873f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-98a394-d995767bd-68hkm" Nov 03 02:05:12.604614 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:05:12.604573 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-98a394-d995767bd-68hkm"] Nov 03 02:05:12.680702 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:05:12.680658 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgq28\" (UniqueName: \"kubernetes.io/projected/bcc4bb00-2f9e-4f38-a331-f725729873f0-kube-api-access-jgq28\") pod \"maas-api-e2e-shared-a-w2-98a394-d995767bd-68hkm\" (UID: \"bcc4bb00-2f9e-4f38-a331-f725729873f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-98a394-d995767bd-68hkm" Nov 03 02:05:12.680923 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:05:12.680732 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcc4bb00-2f9e-4f38-a331-f725729873f0-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-98a394-d995767bd-68hkm\" (UID: \"bcc4bb00-2f9e-4f38-a331-f725729873f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-98a394-d995767bd-68hkm" Nov 03 02:05:12.680923 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:05:12.680814 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bcc4bb00-2f9e-4f38-a331-f725729873f0-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-98a394-d995767bd-68hkm\" (UID: \"bcc4bb00-2f9e-4f38-a331-f725729873f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-98a394-d995767bd-68hkm" Nov 03 02:05:12.680923 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:05:12.680864 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcc4bb00-2f9e-4f38-a331-f725729873f0-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-98a394-d995767bd-68hkm\" (UID: \"bcc4bb00-2f9e-4f38-a331-f725729873f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-98a394-d995767bd-68hkm" Nov 03 02:05:12.680923 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:05:12.680901 5437 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-98a394-serving-cert: secret "maas-api-e2e-shared-a-w2-98a394-serving-cert" not found Nov 03 02:05:12.681123 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:05:12.680981 5437 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bcc4bb00-2f9e-4f38-a331-f725729873f0-maas-api-tls podName:bcc4bb00-2f9e-4f38-a331-f725729873f0 nodeName:}" failed. No retries permitted until 2026-11-03 02:05:13.180964311 +0000 UTC m=+2795.210446583 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bcc4bb00-2f9e-4f38-a331-f725729873f0-maas-api-tls") pod "maas-api-e2e-shared-a-w2-98a394-d995767bd-68hkm" (UID: "bcc4bb00-2f9e-4f38-a331-f725729873f0") : secret "maas-api-e2e-shared-a-w2-98a394-serving-cert" not found Nov 03 02:05:12.681521 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:05:12.681489 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcc4bb00-2f9e-4f38-a331-f725729873f0-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-98a394-d995767bd-68hkm\" (UID: \"bcc4bb00-2f9e-4f38-a331-f725729873f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-98a394-d995767bd-68hkm" Nov 03 02:05:12.683340 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:05:12.683313 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcc4bb00-2f9e-4f38-a331-f725729873f0-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-98a394-d995767bd-68hkm\" (UID: \"bcc4bb00-2f9e-4f38-a331-f725729873f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-98a394-d995767bd-68hkm" Nov 03 02:05:12.739272 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:05:12.739222 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:05:12.745134 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:05:12.745037 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgq28\" (UniqueName: \"kubernetes.io/projected/bcc4bb00-2f9e-4f38-a331-f725729873f0-kube-api-access-jgq28\") pod \"maas-api-e2e-shared-a-w2-98a394-d995767bd-68hkm\" (UID: \"bcc4bb00-2f9e-4f38-a331-f725729873f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-98a394-d995767bd-68hkm" Nov 03 02:05:12.958865 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:05:12.958831 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:05:13.187499 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:05:13.187454 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bcc4bb00-2f9e-4f38-a331-f725729873f0-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-98a394-d995767bd-68hkm\" (UID: \"bcc4bb00-2f9e-4f38-a331-f725729873f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-98a394-d995767bd-68hkm" Nov 03 02:05:13.190882 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:05:13.190843 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bcc4bb00-2f9e-4f38-a331-f725729873f0-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-98a394-d995767bd-68hkm\" (UID: \"bcc4bb00-2f9e-4f38-a331-f725729873f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-98a394-d995767bd-68hkm" Nov 03 02:05:13.411930 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:05:13.411894 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-98a394-d995767bd-68hkm" Nov 03 02:05:13.788994 ip-10-0-136-16 kubenswrapper[5437]: W1103 02:05:13.788956 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcc4bb00_2f9e_4f38_a331_f725729873f0.slice/crio-e2e8545a1016d428f7c486bf343c9afcb40f46659f5c8078bdbf368a5b1a192b WatchSource:0}: Error finding container e2e8545a1016d428f7c486bf343c9afcb40f46659f5c8078bdbf368a5b1a192b: Status 404 returned error can't find the container with id e2e8545a1016d428f7c486bf343c9afcb40f46659f5c8078bdbf368a5b1a192b Nov 03 02:05:13.791647 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:05:13.791628 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-98a394-d995767bd-68hkm"] Nov 03 02:05:14.036605 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:05:14.036518 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 02:05:14.186078 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:05:14.186035 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-98a394-d995767bd-68hkm" event={"ID":"bcc4bb00-2f9e-4f38-a331-f725729873f0","Type":"ContainerStarted","Data":"e2e8545a1016d428f7c486bf343c9afcb40f46659f5c8078bdbf368a5b1a192b"} Nov 03 02:05:15.194124 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:05:15.194088 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-98a394-d995767bd-68hkm" event={"ID":"bcc4bb00-2f9e-4f38-a331-f725729873f0","Type":"ContainerStarted","Data":"23f4373d4bdc45e4e6e34f427c8f63eb078cb18e181e48a00bd191cf3b4877cd"} Nov 03 02:05:15.194613 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:05:15.194226 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-98a394-d995767bd-68hkm" Nov 03 02:05:15.315693 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:05:15.315637 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-98a394-d995767bd-68hkm" podStartSLOduration=2.377209155 podStartE2EDuration="3.315623793s" totalImagesPullingTime="938.414638ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:05:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:05:15.314918698 +0000 UTC m=+2797.344400997" watchObservedRunningTime="2026-11-03 02:05:15.315623793 +0000 UTC m=+2797.345106087" Nov 03 02:05:17.604449 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:05:17.604327 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:05:21.207566 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:05:21.207533 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-98a394-d995767bd-68hkm" Nov 03 02:05:22.496757 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:05:22.496714 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:05:27.283902 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:05:27.283863 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:05:32.112284 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:05:32.112246 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:05:32.703292 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:05:32.703254 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:05:33.161722 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:05:33.161690 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:05:33.856251 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:05:33.856216 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:05:34.111215 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:05:34.111122 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:05:37.462420 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:05:37.462379 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:05:39.150572 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:05:39.150532 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:05:39.384211 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:05:39.384174 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:05:41.076759 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:05:41.076708 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:05:42.442103 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:05:42.442050 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:05:42.675683 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:05:42.675640 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:05:44.037150 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:05:44.037110 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 02:05:44.762292 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:05:44.762257 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-praxis-a-b22fcc/praxis-route-b22fcc-kserve-68b8b8c8f7-rcxhx"] Nov 03 02:05:44.766586 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:05:44.766562 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-b22fcc/praxis-route-b22fcc-kserve-68b8b8c8f7-rcxhx" Nov 03 02:05:44.820825 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:05:44.820792 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-b22fcc/praxis-route-b22fcc-kserve-68b8b8c8f7-rcxhx"] Nov 03 02:05:44.869531 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:05:44.869494 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-b22fcc\"/\"kube-root-ca.crt\"" Nov 03 02:05:44.872225 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:05:44.872193 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-b22fcc\"/\"openshift-service-ca.crt\"" Nov 03 02:05:44.872225 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:05:44.872218 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-b22fcc\"/\"praxis-route-b22fcc-kserve-self-signed-certs\"" Nov 03 02:05:44.872417 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:05:44.872220 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-b22fcc\"/\"default-dockercfg-sh58d\"" Nov 03 02:05:44.902262 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:05:44.902225 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvmjl\" (UniqueName: \"kubernetes.io/projected/7fad0e67-4c39-44f3-b199-fb7803ac9ae3-kube-api-access-xvmjl\") pod \"praxis-route-b22fcc-kserve-68b8b8c8f7-rcxhx\" (UID: \"7fad0e67-4c39-44f3-b199-fb7803ac9ae3\") " pod="ai-tenant-e2e-praxis-a-b22fcc/praxis-route-b22fcc-kserve-68b8b8c8f7-rcxhx" Nov 03 02:05:44.902680 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:05:44.902656 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7fad0e67-4c39-44f3-b199-fb7803ac9ae3-model-cache\") pod \"praxis-route-b22fcc-kserve-68b8b8c8f7-rcxhx\" (UID: \"7fad0e67-4c39-44f3-b199-fb7803ac9ae3\") " pod="ai-tenant-e2e-praxis-a-b22fcc/praxis-route-b22fcc-kserve-68b8b8c8f7-rcxhx" Nov 03 02:05:44.902976 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:05:44.902946 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fad0e67-4c39-44f3-b199-fb7803ac9ae3-tls-certs\") pod \"praxis-route-b22fcc-kserve-68b8b8c8f7-rcxhx\" (UID: \"7fad0e67-4c39-44f3-b199-fb7803ac9ae3\") " pod="ai-tenant-e2e-praxis-a-b22fcc/praxis-route-b22fcc-kserve-68b8b8c8f7-rcxhx" Nov 03 02:05:44.903166 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:05:44.903064 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7fad0e67-4c39-44f3-b199-fb7803ac9ae3-dshm\") pod \"praxis-route-b22fcc-kserve-68b8b8c8f7-rcxhx\" (UID: \"7fad0e67-4c39-44f3-b199-fb7803ac9ae3\") " pod="ai-tenant-e2e-praxis-a-b22fcc/praxis-route-b22fcc-kserve-68b8b8c8f7-rcxhx" Nov 03 02:05:44.903166 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:05:44.903138 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fad0e67-4c39-44f3-b199-fb7803ac9ae3-tmp-dir\") pod \"praxis-route-b22fcc-kserve-68b8b8c8f7-rcxhx\" (UID: \"7fad0e67-4c39-44f3-b199-fb7803ac9ae3\") " pod="ai-tenant-e2e-praxis-a-b22fcc/praxis-route-b22fcc-kserve-68b8b8c8f7-rcxhx" Nov 03 02:05:44.904475 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:05:44.903201 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7fad0e67-4c39-44f3-b199-fb7803ac9ae3-home\") pod \"praxis-route-b22fcc-kserve-68b8b8c8f7-rcxhx\" (UID: \"7fad0e67-4c39-44f3-b199-fb7803ac9ae3\") " pod="ai-tenant-e2e-praxis-a-b22fcc/praxis-route-b22fcc-kserve-68b8b8c8f7-rcxhx" Nov 03 02:05:44.907561 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:05:44.907533 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:05:45.004769 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:05:45.004737 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7fad0e67-4c39-44f3-b199-fb7803ac9ae3-dshm\") pod \"praxis-route-b22fcc-kserve-68b8b8c8f7-rcxhx\" (UID: \"7fad0e67-4c39-44f3-b199-fb7803ac9ae3\") " pod="ai-tenant-e2e-praxis-a-b22fcc/praxis-route-b22fcc-kserve-68b8b8c8f7-rcxhx" Nov 03 02:05:45.004968 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:05:45.004805 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fad0e67-4c39-44f3-b199-fb7803ac9ae3-tmp-dir\") pod \"praxis-route-b22fcc-kserve-68b8b8c8f7-rcxhx\" (UID: \"7fad0e67-4c39-44f3-b199-fb7803ac9ae3\") " pod="ai-tenant-e2e-praxis-a-b22fcc/praxis-route-b22fcc-kserve-68b8b8c8f7-rcxhx" Nov 03 02:05:45.004968 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:05:45.004856 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7fad0e67-4c39-44f3-b199-fb7803ac9ae3-home\") pod \"praxis-route-b22fcc-kserve-68b8b8c8f7-rcxhx\" (UID: \"7fad0e67-4c39-44f3-b199-fb7803ac9ae3\") " pod="ai-tenant-e2e-praxis-a-b22fcc/praxis-route-b22fcc-kserve-68b8b8c8f7-rcxhx" Nov 03 02:05:45.004968 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:05:45.004908 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvmjl\" (UniqueName: \"kubernetes.io/projected/7fad0e67-4c39-44f3-b199-fb7803ac9ae3-kube-api-access-xvmjl\") pod \"praxis-route-b22fcc-kserve-68b8b8c8f7-rcxhx\" (UID: \"7fad0e67-4c39-44f3-b199-fb7803ac9ae3\") " pod="ai-tenant-e2e-praxis-a-b22fcc/praxis-route-b22fcc-kserve-68b8b8c8f7-rcxhx" Nov 03 02:05:45.005095 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:05:45.004975 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7fad0e67-4c39-44f3-b199-fb7803ac9ae3-model-cache\") pod \"praxis-route-b22fcc-kserve-68b8b8c8f7-rcxhx\" (UID: \"7fad0e67-4c39-44f3-b199-fb7803ac9ae3\") " pod="ai-tenant-e2e-praxis-a-b22fcc/praxis-route-b22fcc-kserve-68b8b8c8f7-rcxhx" Nov 03 02:05:45.005095 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:05:45.005013 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fad0e67-4c39-44f3-b199-fb7803ac9ae3-tls-certs\") pod \"praxis-route-b22fcc-kserve-68b8b8c8f7-rcxhx\" (UID: \"7fad0e67-4c39-44f3-b199-fb7803ac9ae3\") " pod="ai-tenant-e2e-praxis-a-b22fcc/praxis-route-b22fcc-kserve-68b8b8c8f7-rcxhx" Nov 03 02:05:45.005284 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:05:45.005251 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fad0e67-4c39-44f3-b199-fb7803ac9ae3-tmp-dir\") pod \"praxis-route-b22fcc-kserve-68b8b8c8f7-rcxhx\" (UID: \"7fad0e67-4c39-44f3-b199-fb7803ac9ae3\") " pod="ai-tenant-e2e-praxis-a-b22fcc/praxis-route-b22fcc-kserve-68b8b8c8f7-rcxhx" Nov 03 02:05:45.005545 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:05:45.005525 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7fad0e67-4c39-44f3-b199-fb7803ac9ae3-home\") pod \"praxis-route-b22fcc-kserve-68b8b8c8f7-rcxhx\" (UID: \"7fad0e67-4c39-44f3-b199-fb7803ac9ae3\") " pod="ai-tenant-e2e-praxis-a-b22fcc/praxis-route-b22fcc-kserve-68b8b8c8f7-rcxhx" Nov 03 02:05:45.005776 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:05:45.005750 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7fad0e67-4c39-44f3-b199-fb7803ac9ae3-model-cache\") pod \"praxis-route-b22fcc-kserve-68b8b8c8f7-rcxhx\" (UID: \"7fad0e67-4c39-44f3-b199-fb7803ac9ae3\") " pod="ai-tenant-e2e-praxis-a-b22fcc/praxis-route-b22fcc-kserve-68b8b8c8f7-rcxhx" Nov 03 02:05:45.007281 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:05:45.007256 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7fad0e67-4c39-44f3-b199-fb7803ac9ae3-dshm\") pod \"praxis-route-b22fcc-kserve-68b8b8c8f7-rcxhx\" (UID: \"7fad0e67-4c39-44f3-b199-fb7803ac9ae3\") " pod="ai-tenant-e2e-praxis-a-b22fcc/praxis-route-b22fcc-kserve-68b8b8c8f7-rcxhx" Nov 03 02:05:45.007593 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:05:45.007576 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fad0e67-4c39-44f3-b199-fb7803ac9ae3-tls-certs\") pod \"praxis-route-b22fcc-kserve-68b8b8c8f7-rcxhx\" (UID: \"7fad0e67-4c39-44f3-b199-fb7803ac9ae3\") " pod="ai-tenant-e2e-praxis-a-b22fcc/praxis-route-b22fcc-kserve-68b8b8c8f7-rcxhx" Nov 03 02:05:45.068322 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:05:45.068285 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvmjl\" (UniqueName: \"kubernetes.io/projected/7fad0e67-4c39-44f3-b199-fb7803ac9ae3-kube-api-access-xvmjl\") pod \"praxis-route-b22fcc-kserve-68b8b8c8f7-rcxhx\" (UID: \"7fad0e67-4c39-44f3-b199-fb7803ac9ae3\") " pod="ai-tenant-e2e-praxis-a-b22fcc/praxis-route-b22fcc-kserve-68b8b8c8f7-rcxhx" Nov 03 02:05:45.080216 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:05:45.080187 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-b22fcc/praxis-route-b22fcc-kserve-68b8b8c8f7-rcxhx" Nov 03 02:05:45.454358 ip-10-0-136-16 kubenswrapper[5437]: W1103 02:05:45.454317 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7fad0e67_4c39_44f3_b199_fb7803ac9ae3.slice/crio-b5b7be2b2db3c1c63c216f7ac064be3d6e3aacdf819a9059f1c8e2258cae7316 WatchSource:0}: Error finding container b5b7be2b2db3c1c63c216f7ac064be3d6e3aacdf819a9059f1c8e2258cae7316: Status 404 returned error can't find the container with id b5b7be2b2db3c1c63c216f7ac064be3d6e3aacdf819a9059f1c8e2258cae7316 Nov 03 02:05:45.458046 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:05:45.458012 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-b22fcc/praxis-route-b22fcc-kserve-68b8b8c8f7-rcxhx"] Nov 03 02:05:46.408051 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:05:46.408008 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-b22fcc/praxis-route-b22fcc-kserve-68b8b8c8f7-rcxhx" event={"ID":"7fad0e67-4c39-44f3-b199-fb7803ac9ae3","Type":"ContainerStarted","Data":"540f3cd6141ff372a9bfeb4b7e283dd97f7ddccef4680f931bcf2a97db105dd2"} Nov 03 02:05:46.410352 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:05:46.408058 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-b22fcc/praxis-route-b22fcc-kserve-68b8b8c8f7-rcxhx" event={"ID":"7fad0e67-4c39-44f3-b199-fb7803ac9ae3","Type":"ContainerStarted","Data":"b5b7be2b2db3c1c63c216f7ac064be3d6e3aacdf819a9059f1c8e2258cae7316"} Nov 03 02:05:46.532226 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:05:46.532160 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-praxis-a-b22fcc/praxis-route-b22fcc-kserve-68b8b8c8f7-rcxhx" podStartSLOduration=2.532144159 podStartE2EDuration="2.532144159s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:05:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:05:46.530878651 +0000 UTC m=+2828.560360945" watchObservedRunningTime="2026-11-03 02:05:46.532144159 +0000 UTC m=+2828.561626462" Nov 03 02:05:47.080881 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:05:47.080846 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-praxis-a-b22fcc/praxis-route-b22fcc-kserve-68b8b8c8f7-rcxhx" Nov 03 02:05:47.853270 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:05:47.853226 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:05:49.353747 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:05:49.353630 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:05:49.903886 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:05:49.903844 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:05:50.915129 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:05:50.915093 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:05:51.134392 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:05:51.117593 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:05:51.581069 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:05:51.581033 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:05:51.869846 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:05:51.869740 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:05:52.985346 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:05:52.985304 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:05:55.080359 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:05:55.080312 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-praxis-a-b22fcc/praxis-route-b22fcc-kserve-68b8b8c8f7-rcxhx" Nov 03 02:05:55.093712 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:05:55.093680 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-praxis-a-b22fcc/praxis-route-b22fcc-kserve-68b8b8c8f7-rcxhx" Nov 03 02:05:55.328497 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:05:55.328434 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:05:55.512488 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:05:55.512347 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-praxis-a-b22fcc/praxis-route-b22fcc-kserve-68b8b8c8f7-rcxhx" Nov 03 02:05:55.536062 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:05:55.536018 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:05:56.644796 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:05:56.644751 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:05:57.512542 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:05:57.512443 5437 generic.go:356] "Generic (PLEG): container finished" podID="80edf8e3-195e-4ccf-b826-27adbf5504ef" containerID="29ac49292258890b95a12c62bf5b229da77fa6281b14f9711a199a784f3f8876" exitCode=0 Nov 03 02:05:57.512542 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:05:57.512525 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-98a394-openshift-default-7dff487c9f-rwl2t" event={"ID":"80edf8e3-195e-4ccf-b826-27adbf5504ef","Type":"ContainerDied","Data":"29ac49292258890b95a12c62bf5b229da77fa6281b14f9711a199a784f3f8876"} Nov 03 02:05:57.513001 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:05:57.512984 5437 scope.go:122] "RemoveContainer" containerID="29ac49292258890b95a12c62bf5b229da77fa6281b14f9711a199a784f3f8876" Nov 03 02:05:57.514610 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:05:57.514578 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 02:05:57.514715 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:05:57.514653 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 02:05:57.514715 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:05:57.514685 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 02:05:58.053004 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:05:58.052963 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:05:58.476141 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:05:58.476052 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:05:58.567387 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:05:58.563051 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-98a394-openshift-default-7dff487c9f-rwl2t" event={"ID":"80edf8e3-195e-4ccf-b826-27adbf5504ef","Type":"ContainerStarted","Data":"dea919d716e0cfeb35ccbd912a36d53367a04691c9eddabf4764795ad7674381"} Nov 03 02:05:58.681097 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:05:58.681061 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:05:58.733464 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:05:58.732657 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-98a394-openshift-default-7dff487c9f-rwl2t" Nov 03 02:05:58.733464 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:05:58.732820 5437 patch_prober.go:29] interesting pod/e2e-shared-a-w2-98a394-openshift-default-7dff487c9f-rwl2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Nov 03 02:05:58.733464 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:05:58.732887 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-98a394-openshift-default-7dff487c9f-rwl2t" podUID="80edf8e3-195e-4ccf-b826-27adbf5504ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Nov 03 02:05:59.733505 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:05:59.732772 5437 patch_prober.go:29] interesting pod/e2e-shared-a-w2-98a394-openshift-default-7dff487c9f-rwl2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Nov 03 02:05:59.733505 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:05:59.732839 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-98a394-openshift-default-7dff487c9f-rwl2t" podUID="80edf8e3-195e-4ccf-b826-27adbf5504ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Nov 03 02:06:00.737780 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:06:00.737505 5437 patch_prober.go:29] interesting pod/e2e-shared-a-w2-98a394-openshift-default-7dff487c9f-rwl2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Nov 03 02:06:00.737780 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:06:00.737571 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-98a394-openshift-default-7dff487c9f-rwl2t" podUID="80edf8e3-195e-4ccf-b826-27adbf5504ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Nov 03 02:06:00.744204 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:06:00.744166 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-98a394-openshift-default-7dff487c9f-rwl2t" Nov 03 02:06:01.736390 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:06:01.735854 5437 patch_prober.go:29] interesting pod/e2e-shared-a-w2-98a394-openshift-default-7dff487c9f-rwl2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Nov 03 02:06:01.736390 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:06:01.735927 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-98a394-openshift-default-7dff487c9f-rwl2t" podUID="80edf8e3-195e-4ccf-b826-27adbf5504ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Nov 03 02:06:02.739391 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:06:02.738845 5437 patch_prober.go:29] interesting pod/e2e-shared-a-w2-98a394-openshift-default-7dff487c9f-rwl2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Nov 03 02:06:02.739391 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:06:02.738922 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-98a394-openshift-default-7dff487c9f-rwl2t" podUID="80edf8e3-195e-4ccf-b826-27adbf5504ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Nov 03 02:06:03.456046 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:06:03.455997 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:06:03.730531 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:06:03.730438 5437 patch_prober.go:29] interesting pod/e2e-shared-a-w2-98a394-openshift-default-7dff487c9f-rwl2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Nov 03 02:06:03.730531 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:06:03.730498 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-98a394-openshift-default-7dff487c9f-rwl2t" podUID="80edf8e3-195e-4ccf-b826-27adbf5504ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Nov 03 02:06:04.734839 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:06:04.734803 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-98a394-openshift-default-7dff487c9f-rwl2t" Nov 03 02:06:04.736007 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:06:04.735981 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-98a394-openshift-default-7dff487c9f-rwl2t" Nov 03 02:06:08.546707 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:06:08.546664 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:06:11.002338 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:06:11.002298 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:06:11.295327 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:06:11.295226 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:06:11.500017 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:06:11.499968 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:06:13.635323 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:06:13.635283 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:06:14.037726 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:06:14.037638 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 02:06:15.436853 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:06:15.436812 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:06:15.651091 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:06:15.651056 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:06:18.586340 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:06:18.586300 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:06:18.838987 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:06:18.838890 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:06:19.093812 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:06:19.093722 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-d9576a-5556d5f859-sr8mk"] Nov 03 02:06:19.097251 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:06:19.097224 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-d9576a-5556d5f859-sr8mk" Nov 03 02:06:19.153795 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:06:19.153757 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-d9576a-5556d5f859-sr8mk"] Nov 03 02:06:19.162272 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:06:19.162238 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6050630e-79eb-49c9-bb97-4cd9e9309ffa-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-d9576a-5556d5f859-sr8mk\" (UID: \"6050630e-79eb-49c9-bb97-4cd9e9309ffa\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-d9576a-5556d5f859-sr8mk" Nov 03 02:06:19.162464 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:06:19.162288 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k42x6\" (UniqueName: \"kubernetes.io/projected/6050630e-79eb-49c9-bb97-4cd9e9309ffa-kube-api-access-k42x6\") pod \"payload-processing-e2e-shared-b-w4-d9576a-5556d5f859-sr8mk\" (UID: \"6050630e-79eb-49c9-bb97-4cd9e9309ffa\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-d9576a-5556d5f859-sr8mk" Nov 03 02:06:19.201150 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:06:19.201104 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-d9576a\"" Nov 03 02:06:19.201150 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:06:19.201115 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-d9576a-dockercfg-k8xlv\"" Nov 03 02:06:19.263660 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:06:19.263622 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6050630e-79eb-49c9-bb97-4cd9e9309ffa-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-d9576a-5556d5f859-sr8mk\" (UID: \"6050630e-79eb-49c9-bb97-4cd9e9309ffa\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-d9576a-5556d5f859-sr8mk" Nov 03 02:06:19.263864 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:06:19.263674 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k42x6\" (UniqueName: \"kubernetes.io/projected/6050630e-79eb-49c9-bb97-4cd9e9309ffa-kube-api-access-k42x6\") pod \"payload-processing-e2e-shared-b-w4-d9576a-5556d5f859-sr8mk\" (UID: \"6050630e-79eb-49c9-bb97-4cd9e9309ffa\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-d9576a-5556d5f859-sr8mk" Nov 03 02:06:19.264471 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:06:19.264450 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6050630e-79eb-49c9-bb97-4cd9e9309ffa-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-d9576a-5556d5f859-sr8mk\" (UID: \"6050630e-79eb-49c9-bb97-4cd9e9309ffa\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-d9576a-5556d5f859-sr8mk" Nov 03 02:06:19.327290 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:06:19.327254 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k42x6\" (UniqueName: \"kubernetes.io/projected/6050630e-79eb-49c9-bb97-4cd9e9309ffa-kube-api-access-k42x6\") pod \"payload-processing-e2e-shared-b-w4-d9576a-5556d5f859-sr8mk\" (UID: \"6050630e-79eb-49c9-bb97-4cd9e9309ffa\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-d9576a-5556d5f859-sr8mk" Nov 03 02:06:19.410431 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:06:19.410317 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-d9576a-5556d5f859-sr8mk" Nov 03 02:06:19.785126 ip-10-0-136-16 kubenswrapper[5437]: W1103 02:06:19.785091 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6050630e_79eb_49c9_bb97_4cd9e9309ffa.slice/crio-2e62d452339cc19155c6c71cf6898590f5598946a12a6fd7a68eac23924a5e27 WatchSource:0}: Error finding container 2e62d452339cc19155c6c71cf6898590f5598946a12a6fd7a68eac23924a5e27: Status 404 returned error can't find the container with id 2e62d452339cc19155c6c71cf6898590f5598946a12a6fd7a68eac23924a5e27 Nov 03 02:06:19.788540 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:06:19.788513 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-d9576a-5556d5f859-sr8mk"] Nov 03 02:06:20.711467 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:06:20.711424 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-d9576a-5556d5f859-sr8mk" event={"ID":"6050630e-79eb-49c9-bb97-4cd9e9309ffa","Type":"ContainerStarted","Data":"ceb93c34598d433b93aa473eb655c7a5941ee64b09c5002f23159d560191ec41"} Nov 03 02:06:20.711922 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:06:20.711522 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-d9576a-5556d5f859-sr8mk" event={"ID":"6050630e-79eb-49c9-bb97-4cd9e9309ffa","Type":"ContainerStarted","Data":"2e62d452339cc19155c6c71cf6898590f5598946a12a6fd7a68eac23924a5e27"} Nov 03 02:06:20.711922 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:06:20.711558 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-d9576a-5556d5f859-sr8mk" Nov 03 02:06:20.833103 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:06:20.833042 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-d9576a-5556d5f859-sr8mk" podStartSLOduration=1.833029583 podStartE2EDuration="1.833029583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:06:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:06:20.830775587 +0000 UTC m=+2862.860257882" watchObservedRunningTime="2026-11-03 02:06:20.833029583 +0000 UTC m=+2862.862511926" Nov 03 02:06:23.863044 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:06:23.863010 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:06:24.925941 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:06:24.925893 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:06:25.410080 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:06:25.410044 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:06:26.035187 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:06:26.035155 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:06:26.334311 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:06:26.334273 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:06:28.866305 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:06:28.866264 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:06:30.683566 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:06:30.683452 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:06:30.904353 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:06:30.904310 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:06:31.722777 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:06:31.722746 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-d9576a-5556d5f859-sr8mk" Nov 03 02:06:33.808508 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:06:33.808467 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:06:34.014770 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:06:34.014738 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:06:34.156340 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:06:34.156251 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6a5d73-78cd98899-wxsl9"] Nov 03 02:06:34.160790 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:06:34.160767 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6a5d73-78cd98899-wxsl9" Nov 03 02:06:34.215407 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:06:34.215374 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-6a5d73\"" Nov 03 02:06:34.215680 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:06:34.215497 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-6a5d73-dockercfg-2zt4z\"" Nov 03 02:06:34.216457 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:06:34.216435 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6a5d73-78cd98899-wxsl9"] Nov 03 02:06:34.220952 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:06:34.220917 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:06:34.265915 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:06:34.265873 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5d7ce00-728b-437c-9fbd-524d8c4697fc-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-6a5d73-78cd98899-wxsl9\" (UID: \"b5d7ce00-728b-437c-9fbd-524d8c4697fc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6a5d73-78cd98899-wxsl9" Nov 03 02:06:34.266142 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:06:34.265956 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzztm\" (UniqueName: \"kubernetes.io/projected/b5d7ce00-728b-437c-9fbd-524d8c4697fc-kube-api-access-mzztm\") pod \"payload-pre-processing-e2e-shared-b-w2-6a5d73-78cd98899-wxsl9\" (UID: \"b5d7ce00-728b-437c-9fbd-524d8c4697fc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6a5d73-78cd98899-wxsl9" Nov 03 02:06:34.320239 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:06:34.320170 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-6a5d73-695fb45cff-zbh87"] Nov 03 02:06:34.329146 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:06:34.329111 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6a5d73-695fb45cff-zbh87" Nov 03 02:06:34.367544 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:06:34.367496 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5d7ce00-728b-437c-9fbd-524d8c4697fc-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-6a5d73-78cd98899-wxsl9\" (UID: \"b5d7ce00-728b-437c-9fbd-524d8c4697fc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6a5d73-78cd98899-wxsl9" Nov 03 02:06:34.367718 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:06:34.367566 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzztm\" (UniqueName: \"kubernetes.io/projected/b5d7ce00-728b-437c-9fbd-524d8c4697fc-kube-api-access-mzztm\") pod \"payload-pre-processing-e2e-shared-b-w2-6a5d73-78cd98899-wxsl9\" (UID: \"b5d7ce00-728b-437c-9fbd-524d8c4697fc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6a5d73-78cd98899-wxsl9" Nov 03 02:06:34.367718 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:06:34.367613 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8374d59d-7bed-4b5d-b887-013569604481-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-6a5d73-695fb45cff-zbh87\" (UID: \"8374d59d-7bed-4b5d-b887-013569604481\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6a5d73-695fb45cff-zbh87" Nov 03 02:06:34.367718 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:06:34.367696 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vsmc\" (UniqueName: \"kubernetes.io/projected/8374d59d-7bed-4b5d-b887-013569604481-kube-api-access-8vsmc\") pod \"payload-processing-e2e-shared-b-w2-6a5d73-695fb45cff-zbh87\" (UID: \"8374d59d-7bed-4b5d-b887-013569604481\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6a5d73-695fb45cff-zbh87" Nov 03 02:06:34.368765 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:06:34.368734 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5d7ce00-728b-437c-9fbd-524d8c4697fc-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-6a5d73-78cd98899-wxsl9\" (UID: \"b5d7ce00-728b-437c-9fbd-524d8c4697fc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6a5d73-78cd98899-wxsl9" Nov 03 02:06:34.373833 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:06:34.373810 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-6a5d73-695fb45cff-zbh87"] Nov 03 02:06:34.431134 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:06:34.431056 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzztm\" (UniqueName: \"kubernetes.io/projected/b5d7ce00-728b-437c-9fbd-524d8c4697fc-kube-api-access-mzztm\") pod \"payload-pre-processing-e2e-shared-b-w2-6a5d73-78cd98899-wxsl9\" (UID: \"b5d7ce00-728b-437c-9fbd-524d8c4697fc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6a5d73-78cd98899-wxsl9" Nov 03 02:06:34.469562 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:06:34.469510 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vsmc\" (UniqueName: \"kubernetes.io/projected/8374d59d-7bed-4b5d-b887-013569604481-kube-api-access-8vsmc\") pod \"payload-processing-e2e-shared-b-w2-6a5d73-695fb45cff-zbh87\" (UID: \"8374d59d-7bed-4b5d-b887-013569604481\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6a5d73-695fb45cff-zbh87" Nov 03 02:06:34.469776 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:06:34.469705 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8374d59d-7bed-4b5d-b887-013569604481-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-6a5d73-695fb45cff-zbh87\" (UID: \"8374d59d-7bed-4b5d-b887-013569604481\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6a5d73-695fb45cff-zbh87" Nov 03 02:06:34.470441 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:06:34.470420 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8374d59d-7bed-4b5d-b887-013569604481-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-6a5d73-695fb45cff-zbh87\" (UID: \"8374d59d-7bed-4b5d-b887-013569604481\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6a5d73-695fb45cff-zbh87" Nov 03 02:06:34.475126 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:06:34.475104 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6a5d73-78cd98899-wxsl9" Nov 03 02:06:34.533113 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:06:34.533075 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vsmc\" (UniqueName: \"kubernetes.io/projected/8374d59d-7bed-4b5d-b887-013569604481-kube-api-access-8vsmc\") pod \"payload-processing-e2e-shared-b-w2-6a5d73-695fb45cff-zbh87\" (UID: \"8374d59d-7bed-4b5d-b887-013569604481\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6a5d73-695fb45cff-zbh87" Nov 03 02:06:34.653770 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:06:34.653730 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6a5d73-695fb45cff-zbh87" Nov 03 02:06:34.852691 ip-10-0-136-16 kubenswrapper[5437]: W1103 02:06:34.852660 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5d7ce00_728b_437c_9fbd_524d8c4697fc.slice/crio-fcce1a979f7864bca6efa1cd6210dba55a5c5efe387b4d8cd9357fa80544dd6d WatchSource:0}: Error finding container fcce1a979f7864bca6efa1cd6210dba55a5c5efe387b4d8cd9357fa80544dd6d: Status 404 returned error can't find the container with id fcce1a979f7864bca6efa1cd6210dba55a5c5efe387b4d8cd9357fa80544dd6d Nov 03 02:06:34.855738 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:06:34.855710 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6a5d73-78cd98899-wxsl9"] Nov 03 02:06:35.025564 ip-10-0-136-16 kubenswrapper[5437]: W1103 02:06:35.025530 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8374d59d_7bed_4b5d_b887_013569604481.slice/crio-43e642965740ec4b1c61d4e3df32c479d4411cd6321fb38804adc3ce95055033 WatchSource:0}: Error finding container 43e642965740ec4b1c61d4e3df32c479d4411cd6321fb38804adc3ce95055033: Status 404 returned error can't find the container with id 43e642965740ec4b1c61d4e3df32c479d4411cd6321fb38804adc3ce95055033 Nov 03 02:06:35.030149 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:06:35.030120 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-6a5d73-695fb45cff-zbh87"] Nov 03 02:06:35.811522 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:06:35.811480 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6a5d73-695fb45cff-zbh87" event={"ID":"8374d59d-7bed-4b5d-b887-013569604481","Type":"ContainerStarted","Data":"7286af09f0fb4080875634c63936c707d51a488fc6d9033746e21c69574eb205"} Nov 03 02:06:35.811746 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:06:35.811554 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6a5d73-695fb45cff-zbh87" event={"ID":"8374d59d-7bed-4b5d-b887-013569604481","Type":"ContainerStarted","Data":"43e642965740ec4b1c61d4e3df32c479d4411cd6321fb38804adc3ce95055033"} Nov 03 02:06:35.811746 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:06:35.811583 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6a5d73-695fb45cff-zbh87" Nov 03 02:06:35.813190 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:06:35.813162 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6a5d73-78cd98899-wxsl9" event={"ID":"b5d7ce00-728b-437c-9fbd-524d8c4697fc","Type":"ContainerStarted","Data":"776902be7b8953f6468b3a73028beb618ac8ad691c974b45fb6c5bb49015b0a4"} Nov 03 02:06:35.813317 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:06:35.813222 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6a5d73-78cd98899-wxsl9" Nov 03 02:06:35.813317 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:06:35.813249 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6a5d73-78cd98899-wxsl9" event={"ID":"b5d7ce00-728b-437c-9fbd-524d8c4697fc","Type":"ContainerStarted","Data":"fcce1a979f7864bca6efa1cd6210dba55a5c5efe387b4d8cd9357fa80544dd6d"} Nov 03 02:06:35.934855 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:06:35.934800 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6a5d73-695fb45cff-zbh87" podStartSLOduration=1.934786281 podStartE2EDuration="1.934786281s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:06:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:06:35.933736871 +0000 UTC m=+2877.963219166" watchObservedRunningTime="2026-11-03 02:06:35.934786281 +0000 UTC m=+2877.964268576" Nov 03 02:06:36.055400 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:06:36.055325 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6a5d73-78cd98899-wxsl9" podStartSLOduration=2.055310238 podStartE2EDuration="2.055310238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:06:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:06:36.052972135 +0000 UTC m=+2878.082454431" watchObservedRunningTime="2026-11-03 02:06:36.055310238 +0000 UTC m=+2878.084792535" Nov 03 02:06:38.787109 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:06:38.787076 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-434b6a-7d4c599fb6-t9vcs"] Nov 03 02:06:38.791003 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:06:38.790984 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-434b6a-7d4c599fb6-t9vcs" Nov 03 02:06:38.846063 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:06:38.846028 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-434b6a-7d4c599fb6-t9vcs"] Nov 03 02:06:38.894747 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:06:38.894713 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-434b6a-serving-cert\"" Nov 03 02:06:38.933737 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:06:38.933705 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21b04f72-d240-4ebb-860b-c80607e0d94d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-434b6a-7d4c599fb6-t9vcs\" (UID: \"21b04f72-d240-4ebb-860b-c80607e0d94d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-434b6a-7d4c599fb6-t9vcs" Nov 03 02:06:38.933904 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:06:38.933876 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21b04f72-d240-4ebb-860b-c80607e0d94d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-434b6a-7d4c599fb6-t9vcs\" (UID: \"21b04f72-d240-4ebb-860b-c80607e0d94d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-434b6a-7d4c599fb6-t9vcs" Nov 03 02:06:38.933960 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:06:38.933953 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21b04f72-d240-4ebb-860b-c80607e0d94d-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-434b6a-7d4c599fb6-t9vcs\" (UID: \"21b04f72-d240-4ebb-860b-c80607e0d94d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-434b6a-7d4c599fb6-t9vcs" Nov 03 02:06:38.933999 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:06:38.933974 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvndr\" (UniqueName: \"kubernetes.io/projected/21b04f72-d240-4ebb-860b-c80607e0d94d-kube-api-access-tvndr\") pod \"maas-api-e2e-shared-b-w3-434b6a-7d4c599fb6-t9vcs\" (UID: \"21b04f72-d240-4ebb-860b-c80607e0d94d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-434b6a-7d4c599fb6-t9vcs" Nov 03 02:06:39.028000 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:06:39.027959 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:06:39.035414 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:06:39.035359 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21b04f72-d240-4ebb-860b-c80607e0d94d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-434b6a-7d4c599fb6-t9vcs\" (UID: \"21b04f72-d240-4ebb-860b-c80607e0d94d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-434b6a-7d4c599fb6-t9vcs" Nov 03 02:06:39.035562 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:06:39.035522 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21b04f72-d240-4ebb-860b-c80607e0d94d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-434b6a-7d4c599fb6-t9vcs\" (UID: \"21b04f72-d240-4ebb-860b-c80607e0d94d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-434b6a-7d4c599fb6-t9vcs" Nov 03 02:06:39.035611 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:06:39.035602 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21b04f72-d240-4ebb-860b-c80607e0d94d-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-434b6a-7d4c599fb6-t9vcs\" (UID: \"21b04f72-d240-4ebb-860b-c80607e0d94d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-434b6a-7d4c599fb6-t9vcs" Nov 03 02:06:39.035655 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:06:39.035626 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvndr\" (UniqueName: \"kubernetes.io/projected/21b04f72-d240-4ebb-860b-c80607e0d94d-kube-api-access-tvndr\") pod \"maas-api-e2e-shared-b-w3-434b6a-7d4c599fb6-t9vcs\" (UID: \"21b04f72-d240-4ebb-860b-c80607e0d94d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-434b6a-7d4c599fb6-t9vcs" Nov 03 02:06:39.036034 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:06:39.036016 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21b04f72-d240-4ebb-860b-c80607e0d94d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-434b6a-7d4c599fb6-t9vcs\" (UID: \"21b04f72-d240-4ebb-860b-c80607e0d94d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-434b6a-7d4c599fb6-t9vcs" Nov 03 02:06:39.038136 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:06:39.038070 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21b04f72-d240-4ebb-860b-c80607e0d94d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-434b6a-7d4c599fb6-t9vcs\" (UID: \"21b04f72-d240-4ebb-860b-c80607e0d94d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-434b6a-7d4c599fb6-t9vcs" Nov 03 02:06:39.038136 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:06:39.038093 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21b04f72-d240-4ebb-860b-c80607e0d94d-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-434b6a-7d4c599fb6-t9vcs\" (UID: \"21b04f72-d240-4ebb-860b-c80607e0d94d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-434b6a-7d4c599fb6-t9vcs" Nov 03 02:06:39.098456 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:06:39.098423 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvndr\" (UniqueName: \"kubernetes.io/projected/21b04f72-d240-4ebb-860b-c80607e0d94d-kube-api-access-tvndr\") pod \"maas-api-e2e-shared-b-w3-434b6a-7d4c599fb6-t9vcs\" (UID: \"21b04f72-d240-4ebb-860b-c80607e0d94d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-434b6a-7d4c599fb6-t9vcs" Nov 03 02:06:39.104211 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:06:39.104184 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-434b6a-7d4c599fb6-t9vcs" Nov 03 02:06:39.240183 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:06:39.240072 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:06:39.477557 ip-10-0-136-16 kubenswrapper[5437]: W1103 02:06:39.477524 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21b04f72_d240_4ebb_860b_c80607e0d94d.slice/crio-26192d45480c586ae09589db229a62f22e3c92de5d20c7ed365199c2536dceee WatchSource:0}: Error finding container 26192d45480c586ae09589db229a62f22e3c92de5d20c7ed365199c2536dceee: Status 404 returned error can't find the container with id 26192d45480c586ae09589db229a62f22e3c92de5d20c7ed365199c2536dceee Nov 03 02:06:39.481155 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:06:39.481130 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-434b6a-7d4c599fb6-t9vcs"] Nov 03 02:06:39.839805 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:06:39.839763 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-434b6a-7d4c599fb6-t9vcs" event={"ID":"21b04f72-d240-4ebb-860b-c80607e0d94d","Type":"ContainerStarted","Data":"26192d45480c586ae09589db229a62f22e3c92de5d20c7ed365199c2536dceee"} Nov 03 02:06:40.851052 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:06:40.851012 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-434b6a-7d4c599fb6-t9vcs" event={"ID":"21b04f72-d240-4ebb-860b-c80607e0d94d","Type":"ContainerStarted","Data":"e71e8296b2f8c8956645d07aa0e055907cba9399d9a49f8f25ad5808ccfbc03e"} Nov 03 02:06:40.851506 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:06:40.851231 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-434b6a-7d4c599fb6-t9vcs" Nov 03 02:06:40.972944 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:06:40.972891 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-434b6a-7d4c599fb6-t9vcs" podStartSLOduration=2.015949399 podStartE2EDuration="2.972876161s" totalImagesPullingTime="956.926762ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:06:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:06:40.9715505 +0000 UTC m=+2883.001032796" watchObservedRunningTime="2026-11-03 02:06:40.972876161 +0000 UTC m=+2883.002358456" Nov 03 02:06:43.985485 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:06:43.985438 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:06:44.038780 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:06:44.038741 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 02:06:46.822658 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:06:46.822627 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6a5d73-78cd98899-wxsl9" Nov 03 02:06:46.823099 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:06:46.822698 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6a5d73-695fb45cff-zbh87" Nov 03 02:06:46.864251 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:06:46.864177 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-434b6a-7d4c599fb6-t9vcs" Nov 03 02:06:48.811862 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:06:48.811823 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:06:52.222509 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:06:52.222466 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:06:53.582791 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:06:53.582754 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:06:56.986017 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:06:56.985980 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:06:58.367884 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:06:58.367840 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:07:03.009193 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:07:03.009155 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:07:03.212969 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:07:03.212932 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:07:08.012275 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:07:08.012238 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:07:12.789817 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:07:12.789780 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:07:14.036283 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:07:14.036229 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 02:07:17.514029 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:07:17.513993 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:07:22.262121 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:07:22.262078 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:07:26.915917 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:07:26.915875 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:07:27.166619 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:07:27.166524 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:07:27.373896 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:07:27.373851 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:07:31.851925 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:07:31.851877 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:07:36.607104 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:07:36.607024 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:07:36.808753 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:07:36.808709 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:07:38.919619 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:07:38.919580 5437 scope.go:122] "RemoveContainer" containerID="0027529b6afedab89964a2812a63b6aad847fe93416d6ca4dd91a75a4c14bc54" Nov 03 02:07:41.243935 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:07:41.243892 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:07:41.495519 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:07:41.495408 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:07:44.036957 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:07:44.036916 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 02:07:46.202972 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:07:46.202933 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:07:50.991666 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:07:50.991620 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:07:52.555484 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:07:52.555448 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:07:55.791718 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:07:55.791674 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:07:57.363288 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:07:57.363249 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:08:00.570209 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:08:00.570168 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:08:01.740749 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:08:01.740701 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:08:05.964083 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:08:05.964041 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:08:06.991799 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:08:06.991746 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-059b23/auth-test-model-059b23-kserve-797887d945-j9d4h"] Nov 03 02:08:06.996099 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:08:06.996071 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-059b23/auth-test-model-059b23-kserve-797887d945-j9d4h" Nov 03 02:08:07.020456 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:08:07.020415 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf53e145-1d93-42fe-84eb-116775ebb4d4-tmp-dir\") pod \"auth-test-model-059b23-kserve-797887d945-j9d4h\" (UID: \"bf53e145-1d93-42fe-84eb-116775ebb4d4\") " pod="ai-tenant-e2e-shared-a-w3-059b23/auth-test-model-059b23-kserve-797887d945-j9d4h" Nov 03 02:08:07.020734 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:08:07.020710 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf53e145-1d93-42fe-84eb-116775ebb4d4-home\") pod \"auth-test-model-059b23-kserve-797887d945-j9d4h\" (UID: \"bf53e145-1d93-42fe-84eb-116775ebb4d4\") " pod="ai-tenant-e2e-shared-a-w3-059b23/auth-test-model-059b23-kserve-797887d945-j9d4h" Nov 03 02:08:07.020961 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:08:07.020929 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf53e145-1d93-42fe-84eb-116775ebb4d4-model-cache\") pod \"auth-test-model-059b23-kserve-797887d945-j9d4h\" (UID: \"bf53e145-1d93-42fe-84eb-116775ebb4d4\") " pod="ai-tenant-e2e-shared-a-w3-059b23/auth-test-model-059b23-kserve-797887d945-j9d4h" Nov 03 02:08:07.021046 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:08:07.020979 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtqll\" (UniqueName: \"kubernetes.io/projected/bf53e145-1d93-42fe-84eb-116775ebb4d4-kube-api-access-mtqll\") pod \"auth-test-model-059b23-kserve-797887d945-j9d4h\" (UID: \"bf53e145-1d93-42fe-84eb-116775ebb4d4\") " pod="ai-tenant-e2e-shared-a-w3-059b23/auth-test-model-059b23-kserve-797887d945-j9d4h" Nov 03 02:08:07.021046 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:08:07.021041 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf53e145-1d93-42fe-84eb-116775ebb4d4-tls-certs\") pod \"auth-test-model-059b23-kserve-797887d945-j9d4h\" (UID: \"bf53e145-1d93-42fe-84eb-116775ebb4d4\") " pod="ai-tenant-e2e-shared-a-w3-059b23/auth-test-model-059b23-kserve-797887d945-j9d4h" Nov 03 02:08:07.021199 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:08:07.021181 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf53e145-1d93-42fe-84eb-116775ebb4d4-dshm\") pod \"auth-test-model-059b23-kserve-797887d945-j9d4h\" (UID: \"bf53e145-1d93-42fe-84eb-116775ebb4d4\") " pod="ai-tenant-e2e-shared-a-w3-059b23/auth-test-model-059b23-kserve-797887d945-j9d4h" Nov 03 02:08:07.051118 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:08:07.051082 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-059b23/auth-test-model-059b23-kserve-797887d945-j9d4h"] Nov 03 02:08:07.098274 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:08:07.098237 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-059b23\"/\"kube-root-ca.crt\"" Nov 03 02:08:07.098274 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:08:07.098238 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-059b23\"/\"openshift-service-ca.crt\"" Nov 03 02:08:07.111294 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:08:07.111264 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-059b23\"/\"default-dockercfg-ltblp\"" Nov 03 02:08:07.111465 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:08:07.111355 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-059b23\"/\"auth-test-model-059b23-kserve-self-signed-certs\"" Nov 03 02:08:07.122518 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:08:07.122482 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf53e145-1d93-42fe-84eb-116775ebb4d4-dshm\") pod \"auth-test-model-059b23-kserve-797887d945-j9d4h\" (UID: \"bf53e145-1d93-42fe-84eb-116775ebb4d4\") " pod="ai-tenant-e2e-shared-a-w3-059b23/auth-test-model-059b23-kserve-797887d945-j9d4h" Nov 03 02:08:07.122738 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:08:07.122579 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf53e145-1d93-42fe-84eb-116775ebb4d4-tmp-dir\") pod \"auth-test-model-059b23-kserve-797887d945-j9d4h\" (UID: \"bf53e145-1d93-42fe-84eb-116775ebb4d4\") " pod="ai-tenant-e2e-shared-a-w3-059b23/auth-test-model-059b23-kserve-797887d945-j9d4h" Nov 03 02:08:07.122738 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:08:07.122620 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf53e145-1d93-42fe-84eb-116775ebb4d4-home\") pod \"auth-test-model-059b23-kserve-797887d945-j9d4h\" (UID: \"bf53e145-1d93-42fe-84eb-116775ebb4d4\") " pod="ai-tenant-e2e-shared-a-w3-059b23/auth-test-model-059b23-kserve-797887d945-j9d4h" Nov 03 02:08:07.122738 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:08:07.122668 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf53e145-1d93-42fe-84eb-116775ebb4d4-model-cache\") pod \"auth-test-model-059b23-kserve-797887d945-j9d4h\" (UID: \"bf53e145-1d93-42fe-84eb-116775ebb4d4\") " pod="ai-tenant-e2e-shared-a-w3-059b23/auth-test-model-059b23-kserve-797887d945-j9d4h" Nov 03 02:08:07.122738 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:08:07.122699 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtqll\" (UniqueName: \"kubernetes.io/projected/bf53e145-1d93-42fe-84eb-116775ebb4d4-kube-api-access-mtqll\") pod \"auth-test-model-059b23-kserve-797887d945-j9d4h\" (UID: \"bf53e145-1d93-42fe-84eb-116775ebb4d4\") " pod="ai-tenant-e2e-shared-a-w3-059b23/auth-test-model-059b23-kserve-797887d945-j9d4h" Nov 03 02:08:07.122957 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:08:07.122757 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf53e145-1d93-42fe-84eb-116775ebb4d4-tls-certs\") pod \"auth-test-model-059b23-kserve-797887d945-j9d4h\" (UID: \"bf53e145-1d93-42fe-84eb-116775ebb4d4\") " pod="ai-tenant-e2e-shared-a-w3-059b23/auth-test-model-059b23-kserve-797887d945-j9d4h" Nov 03 02:08:07.123073 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:08:07.123049 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf53e145-1d93-42fe-84eb-116775ebb4d4-tmp-dir\") pod \"auth-test-model-059b23-kserve-797887d945-j9d4h\" (UID: \"bf53e145-1d93-42fe-84eb-116775ebb4d4\") " pod="ai-tenant-e2e-shared-a-w3-059b23/auth-test-model-059b23-kserve-797887d945-j9d4h" Nov 03 02:08:07.123129 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:08:07.123086 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf53e145-1d93-42fe-84eb-116775ebb4d4-home\") pod \"auth-test-model-059b23-kserve-797887d945-j9d4h\" (UID: \"bf53e145-1d93-42fe-84eb-116775ebb4d4\") " pod="ai-tenant-e2e-shared-a-w3-059b23/auth-test-model-059b23-kserve-797887d945-j9d4h" Nov 03 02:08:07.123129 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:08:07.123120 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf53e145-1d93-42fe-84eb-116775ebb4d4-model-cache\") pod \"auth-test-model-059b23-kserve-797887d945-j9d4h\" (UID: \"bf53e145-1d93-42fe-84eb-116775ebb4d4\") " pod="ai-tenant-e2e-shared-a-w3-059b23/auth-test-model-059b23-kserve-797887d945-j9d4h" Nov 03 02:08:07.125195 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:08:07.125163 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf53e145-1d93-42fe-84eb-116775ebb4d4-dshm\") pod \"auth-test-model-059b23-kserve-797887d945-j9d4h\" (UID: \"bf53e145-1d93-42fe-84eb-116775ebb4d4\") " pod="ai-tenant-e2e-shared-a-w3-059b23/auth-test-model-059b23-kserve-797887d945-j9d4h" Nov 03 02:08:07.125484 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:08:07.125465 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf53e145-1d93-42fe-84eb-116775ebb4d4-tls-certs\") pod \"auth-test-model-059b23-kserve-797887d945-j9d4h\" (UID: \"bf53e145-1d93-42fe-84eb-116775ebb4d4\") " pod="ai-tenant-e2e-shared-a-w3-059b23/auth-test-model-059b23-kserve-797887d945-j9d4h" Nov 03 02:08:07.140033 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:08:07.139998 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:08:07.186393 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:08:07.186325 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtqll\" (UniqueName: \"kubernetes.io/projected/bf53e145-1d93-42fe-84eb-116775ebb4d4-kube-api-access-mtqll\") pod \"auth-test-model-059b23-kserve-797887d945-j9d4h\" (UID: \"bf53e145-1d93-42fe-84eb-116775ebb4d4\") " pod="ai-tenant-e2e-shared-a-w3-059b23/auth-test-model-059b23-kserve-797887d945-j9d4h" Nov 03 02:08:07.310628 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:08:07.310595 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-059b23/auth-test-model-059b23-kserve-797887d945-j9d4h" Nov 03 02:08:07.686357 ip-10-0-136-16 kubenswrapper[5437]: W1103 02:08:07.686324 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbf53e145_1d93_42fe_84eb_116775ebb4d4.slice/crio-8f47d8e56101c0cb42fee381e62aa6436821037507366a169256e5821a39ebfe WatchSource:0}: Error finding container 8f47d8e56101c0cb42fee381e62aa6436821037507366a169256e5821a39ebfe: Status 404 returned error can't find the container with id 8f47d8e56101c0cb42fee381e62aa6436821037507366a169256e5821a39ebfe Nov 03 02:08:07.689713 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:08:07.689685 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-059b23/auth-test-model-059b23-kserve-797887d945-j9d4h"] Nov 03 02:08:08.489432 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:08:08.489395 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-059b23/auth-test-model-059b23-kserve-797887d945-j9d4h" event={"ID":"bf53e145-1d93-42fe-84eb-116775ebb4d4","Type":"ContainerStarted","Data":"edac66301490ec37937eb8c10dc3ae7cf6c71bb959c63d8cc22049b1e9182e1f"} Nov 03 02:08:08.489432 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:08:08.489435 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-059b23/auth-test-model-059b23-kserve-797887d945-j9d4h" event={"ID":"bf53e145-1d93-42fe-84eb-116775ebb4d4","Type":"ContainerStarted","Data":"8f47d8e56101c0cb42fee381e62aa6436821037507366a169256e5821a39ebfe"} Nov 03 02:08:08.611739 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:08:08.611666 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-059b23/auth-test-model-059b23-kserve-797887d945-j9d4h" podStartSLOduration=2.611646254 podStartE2EDuration="2.611646254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:08:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:08:08.610023314 +0000 UTC m=+2970.639505610" watchObservedRunningTime="2026-11-03 02:08:08.611646254 +0000 UTC m=+2970.641128549" Nov 03 02:08:09.311455 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:08:09.311409 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-059b23/auth-test-model-059b23-kserve-797887d945-j9d4h" Nov 03 02:08:11.071665 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:08:11.071618 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:08:14.084213 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:08:14.084167 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 02:08:15.844840 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:08:15.844782 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:08:17.311052 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:08:17.310968 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-059b23/auth-test-model-059b23-kserve-797887d945-j9d4h" Nov 03 02:08:17.324120 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:08:17.324082 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-059b23/auth-test-model-059b23-kserve-797887d945-j9d4h" Nov 03 02:08:17.562281 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:08:17.562182 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-059b23/auth-test-model-059b23-kserve-797887d945-j9d4h" Nov 03 02:08:18.349532 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:08:18.349492 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:08:20.578137 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:08:20.578087 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:08:23.261942 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:08:23.261896 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-b22fcc/praxis-route-b22fcc-kserve-68b8b8c8f7-rcxhx"] Nov 03 02:08:23.389745 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:08:23.389708 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:08:25.377741 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:08:25.377679 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:08:29.209428 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:08:29.209355 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:08:30.431066 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:08:30.431027 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:08:35.194798 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:08:35.194757 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:08:38.283845 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:08:38.283802 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-praxis-a-b22fcc/praxis-route-b22fcc-kserve-68b8b8c8f7-rcxhx" podUID="7fad0e67-4c39-44f3-b199-fb7803ac9ae3" containerName="main" containerID="cri-o://540f3cd6141ff372a9bfeb4b7e283dd97f7ddccef4680f931bcf2a97db105dd2" gracePeriod=45 Nov 03 02:08:38.562464 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:08:38.562432 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-b22fcc/praxis-route-b22fcc-kserve-68b8b8c8f7-rcxhx" Nov 03 02:08:38.664552 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:08:38.664519 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fad0e67-4c39-44f3-b199-fb7803ac9ae3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fad0e67-4c39-44f3-b199-fb7803ac9ae3-tmp-dir\") pod \"7fad0e67-4c39-44f3-b199-fb7803ac9ae3\" (UID: \"7fad0e67-4c39-44f3-b199-fb7803ac9ae3\") " Nov 03 02:08:38.664752 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:08:38.664609 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7fad0e67-4c39-44f3-b199-fb7803ac9ae3-tls-certs\" (UniqueName: \"kubernetes.io/secret/7fad0e67-4c39-44f3-b199-fb7803ac9ae3-tls-certs\") pod \"7fad0e67-4c39-44f3-b199-fb7803ac9ae3\" (UID: \"7fad0e67-4c39-44f3-b199-fb7803ac9ae3\") " Nov 03 02:08:38.664752 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:08:38.664636 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fad0e67-4c39-44f3-b199-fb7803ac9ae3-kube-api-access-xvmjl\" (UniqueName: \"kubernetes.io/projected/7fad0e67-4c39-44f3-b199-fb7803ac9ae3-kube-api-access-xvmjl\") pod \"7fad0e67-4c39-44f3-b199-fb7803ac9ae3\" (UID: \"7fad0e67-4c39-44f3-b199-fb7803ac9ae3\") " Nov 03 02:08:38.664752 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:08:38.664736 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fad0e67-4c39-44f3-b199-fb7803ac9ae3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7fad0e67-4c39-44f3-b199-fb7803ac9ae3-dshm\") pod \"7fad0e67-4c39-44f3-b199-fb7803ac9ae3\" (UID: \"7fad0e67-4c39-44f3-b199-fb7803ac9ae3\") " Nov 03 02:08:38.664933 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:08:38.664758 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fad0e67-4c39-44f3-b199-fb7803ac9ae3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7fad0e67-4c39-44f3-b199-fb7803ac9ae3-model-cache\") pod \"7fad0e67-4c39-44f3-b199-fb7803ac9ae3\" (UID: \"7fad0e67-4c39-44f3-b199-fb7803ac9ae3\") " Nov 03 02:08:38.664933 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:08:38.664838 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fad0e67-4c39-44f3-b199-fb7803ac9ae3-tmp-dir" pod "7fad0e67-4c39-44f3-b199-fb7803ac9ae3" (UID: "7fad0e67-4c39-44f3-b199-fb7803ac9ae3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:08:38.664933 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:08:38.664878 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fad0e67-4c39-44f3-b199-fb7803ac9ae3-home\" (UniqueName: \"kubernetes.io/empty-dir/7fad0e67-4c39-44f3-b199-fb7803ac9ae3-home\") pod \"7fad0e67-4c39-44f3-b199-fb7803ac9ae3\" (UID: \"7fad0e67-4c39-44f3-b199-fb7803ac9ae3\") " Nov 03 02:08:38.665123 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:08:38.665095 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fad0e67-4c39-44f3-b199-fb7803ac9ae3-model-cache" pod "7fad0e67-4c39-44f3-b199-fb7803ac9ae3" (UID: "7fad0e67-4c39-44f3-b199-fb7803ac9ae3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:08:38.665238 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:08:38.665207 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fad0e67-4c39-44f3-b199-fb7803ac9ae3-home" pod "7fad0e67-4c39-44f3-b199-fb7803ac9ae3" (UID: "7fad0e67-4c39-44f3-b199-fb7803ac9ae3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:08:38.665567 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:08:38.665544 5437 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7fad0e67-4c39-44f3-b199-fb7803ac9ae3-home\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:08:38.665657 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:08:38.665598 5437 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fad0e67-4c39-44f3-b199-fb7803ac9ae3-tmp-dir\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:08:38.665657 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:08:38.665615 5437 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7fad0e67-4c39-44f3-b199-fb7803ac9ae3-model-cache\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:08:38.667022 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:08:38.667000 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fad0e67-4c39-44f3-b199-fb7803ac9ae3-dshm" pod "7fad0e67-4c39-44f3-b199-fb7803ac9ae3" (UID: "7fad0e67-4c39-44f3-b199-fb7803ac9ae3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:08:38.667022 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:08:38.667013 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fad0e67-4c39-44f3-b199-fb7803ac9ae3-tls-certs" pod "7fad0e67-4c39-44f3-b199-fb7803ac9ae3" (UID: "7fad0e67-4c39-44f3-b199-fb7803ac9ae3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:08:38.667161 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:08:38.667141 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fad0e67-4c39-44f3-b199-fb7803ac9ae3-kube-api-access-xvmjl" pod "7fad0e67-4c39-44f3-b199-fb7803ac9ae3" (UID: "7fad0e67-4c39-44f3-b199-fb7803ac9ae3"). InnerVolumeSpecName "kube-api-access-xvmjl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:08:38.718725 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:08:38.718687 5437 generic.go:356] "Generic (PLEG): container finished" podID="7fad0e67-4c39-44f3-b199-fb7803ac9ae3" containerID="540f3cd6141ff372a9bfeb4b7e283dd97f7ddccef4680f931bcf2a97db105dd2" exitCode=0 Nov 03 02:08:38.718894 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:08:38.718745 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-b22fcc/praxis-route-b22fcc-kserve-68b8b8c8f7-rcxhx" event={"ID":"7fad0e67-4c39-44f3-b199-fb7803ac9ae3","Type":"ContainerDied","Data":"540f3cd6141ff372a9bfeb4b7e283dd97f7ddccef4680f931bcf2a97db105dd2"} Nov 03 02:08:38.718894 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:08:38.718787 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-b22fcc/praxis-route-b22fcc-kserve-68b8b8c8f7-rcxhx" event={"ID":"7fad0e67-4c39-44f3-b199-fb7803ac9ae3","Type":"ContainerDied","Data":"b5b7be2b2db3c1c63c216f7ac064be3d6e3aacdf819a9059f1c8e2258cae7316"} Nov 03 02:08:38.718894 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:08:38.718810 5437 scope.go:122] "RemoveContainer" containerID="540f3cd6141ff372a9bfeb4b7e283dd97f7ddccef4680f931bcf2a97db105dd2" Nov 03 02:08:38.718894 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:08:38.718747 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-b22fcc/praxis-route-b22fcc-kserve-68b8b8c8f7-rcxhx" Nov 03 02:08:38.732626 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:08:38.732603 5437 scope.go:122] "RemoveContainer" containerID="540f3cd6141ff372a9bfeb4b7e283dd97f7ddccef4680f931bcf2a97db105dd2" Nov 03 02:08:38.732934 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:08:38.732915 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"540f3cd6141ff372a9bfeb4b7e283dd97f7ddccef4680f931bcf2a97db105dd2\": container with ID starting with 540f3cd6141ff372a9bfeb4b7e283dd97f7ddccef4680f931bcf2a97db105dd2 not found: ID does not exist" containerID="540f3cd6141ff372a9bfeb4b7e283dd97f7ddccef4680f931bcf2a97db105dd2" Nov 03 02:08:38.732985 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:08:38.732944 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"540f3cd6141ff372a9bfeb4b7e283dd97f7ddccef4680f931bcf2a97db105dd2"} err="failed to get container status \"540f3cd6141ff372a9bfeb4b7e283dd97f7ddccef4680f931bcf2a97db105dd2\": rpc error: code = NotFound desc = could not find container \"540f3cd6141ff372a9bfeb4b7e283dd97f7ddccef4680f931bcf2a97db105dd2\": container with ID starting with 540f3cd6141ff372a9bfeb4b7e283dd97f7ddccef4680f931bcf2a97db105dd2 not found: ID does not exist" Nov 03 02:08:38.766725 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:08:38.766696 5437 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fad0e67-4c39-44f3-b199-fb7803ac9ae3-tls-certs\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:08:38.766725 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:08:38.766724 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvmjl\" (UniqueName: \"kubernetes.io/projected/7fad0e67-4c39-44f3-b199-fb7803ac9ae3-kube-api-access-xvmjl\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:08:38.766935 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:08:38.766734 5437 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7fad0e67-4c39-44f3-b199-fb7803ac9ae3-dshm\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:08:38.889864 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:08:38.889828 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-b22fcc/praxis-route-b22fcc-kserve-68b8b8c8f7-rcxhx"] Nov 03 02:08:38.893750 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:08:38.893714 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-praxis-a-b22fcc/praxis-route-b22fcc-kserve-68b8b8c8f7-rcxhx"] Nov 03 02:08:39.499932 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:08:39.499889 5437 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-b22fcc/praxis-route-b22fcc-kserve-68b8b8c8f7-rcxhx" podUID="7fad0e67-4c39-44f3-b199-fb7803ac9ae3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.136:8000/ready\": context deadline exceeded" Nov 03 02:08:39.807106 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:08:39.807052 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:08:40.061542 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:08:40.061452 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:08:40.089653 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:08:40.089616 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7fad0e67-4c39-44f3-b199-fb7803ac9ae3" path="/var/lib/kubelet/pods/7fad0e67-4c39-44f3-b199-fb7803ac9ae3/volumes" Nov 03 02:08:40.260376 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:08:40.260324 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:08:44.039083 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:08:44.039032 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 02:08:44.864824 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:08:44.864781 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:08:49.733478 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:08:49.733437 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:08:52.903354 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:08:52.903311 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:08:54.502706 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:08:54.502655 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:08:54.706035 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:08:54.705991 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:08:57.708137 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:08:57.708097 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:08:59.414769 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:08:59.414721 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:08:59.755765 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:08:59.755680 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:09:00.918816 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:09:00.918775 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:09:01.656899 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:09:01.656855 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:09:01.857763 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:09:01.857723 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:09:04.449020 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:09:04.448958 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:09:06.580531 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:09:06.580498 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:09:09.384347 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:09:09.384299 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:09:09.592959 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:09:09.592912 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:09:12.752201 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:09:12.752162 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:09:13.095734 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:09:13.095692 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w1-a54cab/auth-test-model-a54cab-kserve-7db9d647d9-pddx2"] Nov 03 02:09:13.096399 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:09:13.096379 5437 cpu_manager.go:414] "containerMap: removing container" podUID="7fad0e67-4c39-44f3-b199-fb7803ac9ae3" containerName="main" Nov 03 02:09:13.096456 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:09:13.096403 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fad0e67-4c39-44f3-b199-fb7803ac9ae3" containerName="main" Nov 03 02:09:13.096558 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:09:13.096549 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fad0e67-4c39-44f3-b199-fb7803ac9ae3" containerName="main" Nov 03 02:09:13.100459 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:09:13.100437 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-a54cab/auth-test-model-a54cab-kserve-7db9d647d9-pddx2" Nov 03 02:09:13.155213 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:09:13.155177 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-a54cab/auth-test-model-a54cab-kserve-7db9d647d9-pddx2"] Nov 03 02:09:13.155476 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:09:13.155218 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ee8734a-4bb3-4efd-b601-797918ce5a9f-tls-certs\") pod \"auth-test-model-a54cab-kserve-7db9d647d9-pddx2\" (UID: \"6ee8734a-4bb3-4efd-b601-797918ce5a9f\") " pod="ai-tenant-e2e-shared-a-w1-a54cab/auth-test-model-a54cab-kserve-7db9d647d9-pddx2" Nov 03 02:09:13.155695 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:09:13.155669 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ee8734a-4bb3-4efd-b601-797918ce5a9f-model-cache\") pod \"auth-test-model-a54cab-kserve-7db9d647d9-pddx2\" (UID: \"6ee8734a-4bb3-4efd-b601-797918ce5a9f\") " pod="ai-tenant-e2e-shared-a-w1-a54cab/auth-test-model-a54cab-kserve-7db9d647d9-pddx2" Nov 03 02:09:13.155875 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:09:13.155800 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7k62\" (UniqueName: \"kubernetes.io/projected/6ee8734a-4bb3-4efd-b601-797918ce5a9f-kube-api-access-f7k62\") pod \"auth-test-model-a54cab-kserve-7db9d647d9-pddx2\" (UID: \"6ee8734a-4bb3-4efd-b601-797918ce5a9f\") " pod="ai-tenant-e2e-shared-a-w1-a54cab/auth-test-model-a54cab-kserve-7db9d647d9-pddx2" Nov 03 02:09:13.155944 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:09:13.155881 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ee8734a-4bb3-4efd-b601-797918ce5a9f-home\") pod \"auth-test-model-a54cab-kserve-7db9d647d9-pddx2\" (UID: \"6ee8734a-4bb3-4efd-b601-797918ce5a9f\") " pod="ai-tenant-e2e-shared-a-w1-a54cab/auth-test-model-a54cab-kserve-7db9d647d9-pddx2" Nov 03 02:09:13.156003 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:09:13.155980 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ee8734a-4bb3-4efd-b601-797918ce5a9f-tmp-dir\") pod \"auth-test-model-a54cab-kserve-7db9d647d9-pddx2\" (UID: \"6ee8734a-4bb3-4efd-b601-797918ce5a9f\") " pod="ai-tenant-e2e-shared-a-w1-a54cab/auth-test-model-a54cab-kserve-7db9d647d9-pddx2" Nov 03 02:09:13.156073 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:09:13.156040 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ee8734a-4bb3-4efd-b601-797918ce5a9f-dshm\") pod \"auth-test-model-a54cab-kserve-7db9d647d9-pddx2\" (UID: \"6ee8734a-4bb3-4efd-b601-797918ce5a9f\") " pod="ai-tenant-e2e-shared-a-w1-a54cab/auth-test-model-a54cab-kserve-7db9d647d9-pddx2" Nov 03 02:09:13.202217 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:09:13.202177 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-a54cab\"/\"kube-root-ca.crt\"" Nov 03 02:09:13.202432 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:09:13.202299 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-a54cab\"/\"openshift-service-ca.crt\"" Nov 03 02:09:13.202432 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:09:13.202327 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-a54cab\"/\"default-dockercfg-hjl66\"" Nov 03 02:09:13.202432 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:09:13.202334 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-a54cab\"/\"auth-test-model-a54cab-kserve-self-signed-certs\"" Nov 03 02:09:13.257474 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:09:13.257436 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ee8734a-4bb3-4efd-b601-797918ce5a9f-dshm\") pod \"auth-test-model-a54cab-kserve-7db9d647d9-pddx2\" (UID: \"6ee8734a-4bb3-4efd-b601-797918ce5a9f\") " pod="ai-tenant-e2e-shared-a-w1-a54cab/auth-test-model-a54cab-kserve-7db9d647d9-pddx2" Nov 03 02:09:13.258560 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:09:13.258529 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ee8734a-4bb3-4efd-b601-797918ce5a9f-tls-certs\") pod \"auth-test-model-a54cab-kserve-7db9d647d9-pddx2\" (UID: \"6ee8734a-4bb3-4efd-b601-797918ce5a9f\") " pod="ai-tenant-e2e-shared-a-w1-a54cab/auth-test-model-a54cab-kserve-7db9d647d9-pddx2" Nov 03 02:09:13.258856 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:09:13.258834 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ee8734a-4bb3-4efd-b601-797918ce5a9f-model-cache\") pod \"auth-test-model-a54cab-kserve-7db9d647d9-pddx2\" (UID: \"6ee8734a-4bb3-4efd-b601-797918ce5a9f\") " pod="ai-tenant-e2e-shared-a-w1-a54cab/auth-test-model-a54cab-kserve-7db9d647d9-pddx2" Nov 03 02:09:13.265117 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:09:13.258299 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:09:13.265117 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:09:13.260111 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ee8734a-4bb3-4efd-b601-797918ce5a9f-model-cache\") pod \"auth-test-model-a54cab-kserve-7db9d647d9-pddx2\" (UID: \"6ee8734a-4bb3-4efd-b601-797918ce5a9f\") " pod="ai-tenant-e2e-shared-a-w1-a54cab/auth-test-model-a54cab-kserve-7db9d647d9-pddx2" Nov 03 02:09:13.265641 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:09:13.265604 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7k62\" (UniqueName: \"kubernetes.io/projected/6ee8734a-4bb3-4efd-b601-797918ce5a9f-kube-api-access-f7k62\") pod \"auth-test-model-a54cab-kserve-7db9d647d9-pddx2\" (UID: \"6ee8734a-4bb3-4efd-b601-797918ce5a9f\") " pod="ai-tenant-e2e-shared-a-w1-a54cab/auth-test-model-a54cab-kserve-7db9d647d9-pddx2" Nov 03 02:09:13.265779 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:09:13.265698 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ee8734a-4bb3-4efd-b601-797918ce5a9f-home\") pod \"auth-test-model-a54cab-kserve-7db9d647d9-pddx2\" (UID: \"6ee8734a-4bb3-4efd-b601-797918ce5a9f\") " pod="ai-tenant-e2e-shared-a-w1-a54cab/auth-test-model-a54cab-kserve-7db9d647d9-pddx2" Nov 03 02:09:13.265924 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:09:13.265903 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ee8734a-4bb3-4efd-b601-797918ce5a9f-tmp-dir\") pod \"auth-test-model-a54cab-kserve-7db9d647d9-pddx2\" (UID: \"6ee8734a-4bb3-4efd-b601-797918ce5a9f\") " pod="ai-tenant-e2e-shared-a-w1-a54cab/auth-test-model-a54cab-kserve-7db9d647d9-pddx2" Nov 03 02:09:13.266476 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:09:13.266447 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ee8734a-4bb3-4efd-b601-797918ce5a9f-tmp-dir\") pod \"auth-test-model-a54cab-kserve-7db9d647d9-pddx2\" (UID: \"6ee8734a-4bb3-4efd-b601-797918ce5a9f\") " pod="ai-tenant-e2e-shared-a-w1-a54cab/auth-test-model-a54cab-kserve-7db9d647d9-pddx2" Nov 03 02:09:13.266726 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:09:13.266703 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ee8734a-4bb3-4efd-b601-797918ce5a9f-home\") pod \"auth-test-model-a54cab-kserve-7db9d647d9-pddx2\" (UID: \"6ee8734a-4bb3-4efd-b601-797918ce5a9f\") " pod="ai-tenant-e2e-shared-a-w1-a54cab/auth-test-model-a54cab-kserve-7db9d647d9-pddx2" Nov 03 02:09:13.274167 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:09:13.271539 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ee8734a-4bb3-4efd-b601-797918ce5a9f-tls-certs\") pod \"auth-test-model-a54cab-kserve-7db9d647d9-pddx2\" (UID: \"6ee8734a-4bb3-4efd-b601-797918ce5a9f\") " pod="ai-tenant-e2e-shared-a-w1-a54cab/auth-test-model-a54cab-kserve-7db9d647d9-pddx2" Nov 03 02:09:13.274682 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:09:13.274650 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ee8734a-4bb3-4efd-b601-797918ce5a9f-dshm\") pod \"auth-test-model-a54cab-kserve-7db9d647d9-pddx2\" (UID: \"6ee8734a-4bb3-4efd-b601-797918ce5a9f\") " pod="ai-tenant-e2e-shared-a-w1-a54cab/auth-test-model-a54cab-kserve-7db9d647d9-pddx2" Nov 03 02:09:13.328692 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:09:13.328652 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7k62\" (UniqueName: \"kubernetes.io/projected/6ee8734a-4bb3-4efd-b601-797918ce5a9f-kube-api-access-f7k62\") pod \"auth-test-model-a54cab-kserve-7db9d647d9-pddx2\" (UID: \"6ee8734a-4bb3-4efd-b601-797918ce5a9f\") " pod="ai-tenant-e2e-shared-a-w1-a54cab/auth-test-model-a54cab-kserve-7db9d647d9-pddx2" Nov 03 02:09:13.415605 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:09:13.415497 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-a54cab/auth-test-model-a54cab-kserve-7db9d647d9-pddx2" Nov 03 02:09:13.800014 ip-10-0-136-16 kubenswrapper[5437]: W1103 02:09:13.799981 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6ee8734a_4bb3_4efd_b601_797918ce5a9f.slice/crio-6110c32ad0da13328481618dbffe77106c283e03f4583cd5d9aab4f6c15c900b WatchSource:0}: Error finding container 6110c32ad0da13328481618dbffe77106c283e03f4583cd5d9aab4f6c15c900b: Status 404 returned error can't find the container with id 6110c32ad0da13328481618dbffe77106c283e03f4583cd5d9aab4f6c15c900b Nov 03 02:09:13.803031 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:09:13.803007 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-a54cab/auth-test-model-a54cab-kserve-7db9d647d9-pddx2"] Nov 03 02:09:13.969101 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:09:13.969045 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-a54cab/auth-test-model-a54cab-kserve-7db9d647d9-pddx2" event={"ID":"6ee8734a-4bb3-4efd-b601-797918ce5a9f","Type":"ContainerStarted","Data":"58de8d65e8cf0e8868f3ad3411065da6c01318053dd908475369a70535c00dfe"} Nov 03 02:09:13.969318 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:09:13.969180 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-a54cab/auth-test-model-a54cab-kserve-7db9d647d9-pddx2" event={"ID":"6ee8734a-4bb3-4efd-b601-797918ce5a9f","Type":"ContainerStarted","Data":"6110c32ad0da13328481618dbffe77106c283e03f4583cd5d9aab4f6c15c900b"} Nov 03 02:09:14.038334 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:09:14.038292 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 02:09:14.089932 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:09:14.089873 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w1-a54cab/auth-test-model-a54cab-kserve-7db9d647d9-pddx2" podStartSLOduration=1.089850971 podStartE2EDuration="1.089850971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:09:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:09:14.089162322 +0000 UTC m=+3036.118644619" watchObservedRunningTime="2026-11-03 02:09:14.089850971 +0000 UTC m=+3036.119333266" Nov 03 02:09:14.167555 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:09:14.167517 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:09:14.416507 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:09:14.416411 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-a54cab/auth-test-model-a54cab-kserve-7db9d647d9-pddx2" Nov 03 02:09:18.998093 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:09:18.998054 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:09:23.415951 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:09:23.415910 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w1-a54cab/auth-test-model-a54cab-kserve-7db9d647d9-pddx2" Nov 03 02:09:23.429207 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:09:23.429172 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w1-a54cab/auth-test-model-a54cab-kserve-7db9d647d9-pddx2" Nov 03 02:09:23.877260 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:09:23.877213 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:09:24.050589 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:09:24.050559 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-a54cab/auth-test-model-a54cab-kserve-7db9d647d9-pddx2" Nov 03 02:09:25.943224 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:09:25.943161 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:09:26.144973 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:09:26.144935 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:09:26.408137 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:09:26.408092 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:09:28.580152 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:09:28.580091 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:09:31.973007 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:09:31.972966 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:09:32.177097 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:09:32.177063 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:09:33.375893 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:09:33.375854 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:09:36.565595 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:09:36.565543 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:09:38.167970 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:09:38.167929 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:09:42.916528 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:09:42.916490 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:09:44.035562 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:09:44.035521 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 02:09:47.648953 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:09:47.648794 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:09:52.533909 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:09:52.533847 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:09:53.248989 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:09:53.248952 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:09:53.453671 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:09:53.453634 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:09:57.294352 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:09:57.294305 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:09:58.032482 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:09:58.032440 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:10:02.083876 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:10:02.083832 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:10:04.678336 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:10:04.678297 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-92zjz"] Nov 03 02:10:04.678773 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:10:04.678548 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-92zjz" podUID="748c5d2d-505a-4ac4-8563-d445b5ba48de" containerName="istio-proxy" containerID="cri-o://bb1e63d91d134c2b2812ece736cfcce960f30b8524eedf283f6d09fe493edf2b" gracePeriod=30 Nov 03 02:10:06.879251 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:10:06.879214 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:10:07.753569 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:10:07.753346 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:10:09.963681 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:10:09.963651 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-92zjz" Nov 03 02:10:10.026142 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:10:10.026098 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/748c5d2d-505a-4ac4-8563-d445b5ba48de-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/748c5d2d-505a-4ac4-8563-d445b5ba48de-istio-podinfo\") pod \"748c5d2d-505a-4ac4-8563-d445b5ba48de\" (UID: \"748c5d2d-505a-4ac4-8563-d445b5ba48de\") " Nov 03 02:10:10.026142 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:10:10.026146 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/748c5d2d-505a-4ac4-8563-d445b5ba48de-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/748c5d2d-505a-4ac4-8563-d445b5ba48de-workload-socket\") pod \"748c5d2d-505a-4ac4-8563-d445b5ba48de\" (UID: \"748c5d2d-505a-4ac4-8563-d445b5ba48de\") " Nov 03 02:10:10.026402 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:10:10.026174 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/748c5d2d-505a-4ac4-8563-d445b5ba48de-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/748c5d2d-505a-4ac4-8563-d445b5ba48de-istiod-ca-cert\") pod \"748c5d2d-505a-4ac4-8563-d445b5ba48de\" (UID: \"748c5d2d-505a-4ac4-8563-d445b5ba48de\") " Nov 03 02:10:10.026402 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:10:10.026255 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/748c5d2d-505a-4ac4-8563-d445b5ba48de-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/748c5d2d-505a-4ac4-8563-d445b5ba48de-istio-data\") pod \"748c5d2d-505a-4ac4-8563-d445b5ba48de\" (UID: \"748c5d2d-505a-4ac4-8563-d445b5ba48de\") " Nov 03 02:10:10.026402 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:10:10.026305 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/748c5d2d-505a-4ac4-8563-d445b5ba48de-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/748c5d2d-505a-4ac4-8563-d445b5ba48de-credential-socket\") pod \"748c5d2d-505a-4ac4-8563-d445b5ba48de\" (UID: \"748c5d2d-505a-4ac4-8563-d445b5ba48de\") " Nov 03 02:10:10.026402 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:10:10.026347 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/748c5d2d-505a-4ac4-8563-d445b5ba48de-kube-api-access-72j2b\" (UniqueName: \"kubernetes.io/projected/748c5d2d-505a-4ac4-8563-d445b5ba48de-kube-api-access-72j2b\") pod \"748c5d2d-505a-4ac4-8563-d445b5ba48de\" (UID: \"748c5d2d-505a-4ac4-8563-d445b5ba48de\") " Nov 03 02:10:10.026627 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:10:10.026505 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/748c5d2d-505a-4ac4-8563-d445b5ba48de-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/748c5d2d-505a-4ac4-8563-d445b5ba48de-istio-envoy\") pod \"748c5d2d-505a-4ac4-8563-d445b5ba48de\" (UID: \"748c5d2d-505a-4ac4-8563-d445b5ba48de\") " Nov 03 02:10:10.026627 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:10:10.026543 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/748c5d2d-505a-4ac4-8563-d445b5ba48de-workload-socket" pod "748c5d2d-505a-4ac4-8563-d445b5ba48de" (UID: "748c5d2d-505a-4ac4-8563-d445b5ba48de"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:10:10.026627 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:10:10.026580 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/748c5d2d-505a-4ac4-8563-d445b5ba48de-istio-token\" (UniqueName: \"kubernetes.io/projected/748c5d2d-505a-4ac4-8563-d445b5ba48de-istio-token\") pod \"748c5d2d-505a-4ac4-8563-d445b5ba48de\" (UID: \"748c5d2d-505a-4ac4-8563-d445b5ba48de\") " Nov 03 02:10:10.026627 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:10:10.026616 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/748c5d2d-505a-4ac4-8563-d445b5ba48de-istio-data" pod "748c5d2d-505a-4ac4-8563-d445b5ba48de" (UID: "748c5d2d-505a-4ac4-8563-d445b5ba48de"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:10:10.026827 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:10:10.026634 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/748c5d2d-505a-4ac4-8563-d445b5ba48de-istiod-ca-cert" pod "748c5d2d-505a-4ac4-8563-d445b5ba48de" (UID: "748c5d2d-505a-4ac4-8563-d445b5ba48de"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:10:10.026827 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:10:10.026678 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/748c5d2d-505a-4ac4-8563-d445b5ba48de-credential-socket" pod "748c5d2d-505a-4ac4-8563-d445b5ba48de" (UID: "748c5d2d-505a-4ac4-8563-d445b5ba48de"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:10:10.026827 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:10:10.026725 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/748c5d2d-505a-4ac4-8563-d445b5ba48de-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/748c5d2d-505a-4ac4-8563-d445b5ba48de-workload-certs\") pod \"748c5d2d-505a-4ac4-8563-d445b5ba48de\" (UID: \"748c5d2d-505a-4ac4-8563-d445b5ba48de\") " Nov 03 02:10:10.026997 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:10:10.026907 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/748c5d2d-505a-4ac4-8563-d445b5ba48de-workload-certs" pod "748c5d2d-505a-4ac4-8563-d445b5ba48de" (UID: "748c5d2d-505a-4ac4-8563-d445b5ba48de"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:10:10.027290 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:10:10.027269 5437 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/748c5d2d-505a-4ac4-8563-d445b5ba48de-workload-certs\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:10:10.027290 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:10:10.027291 5437 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/748c5d2d-505a-4ac4-8563-d445b5ba48de-workload-socket\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:10:10.027290 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:10:10.027301 5437 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/748c5d2d-505a-4ac4-8563-d445b5ba48de-istiod-ca-cert\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:10:10.027561 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:10:10.027310 5437 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/748c5d2d-505a-4ac4-8563-d445b5ba48de-istio-data\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:10:10.027561 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:10:10.027321 5437 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/748c5d2d-505a-4ac4-8563-d445b5ba48de-credential-socket\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:10:10.028749 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:10:10.028722 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/748c5d2d-505a-4ac4-8563-d445b5ba48de-istio-envoy" pod "748c5d2d-505a-4ac4-8563-d445b5ba48de" (UID: "748c5d2d-505a-4ac4-8563-d445b5ba48de"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:10:10.028853 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:10:10.028751 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/748c5d2d-505a-4ac4-8563-d445b5ba48de-istio-token" pod "748c5d2d-505a-4ac4-8563-d445b5ba48de" (UID: "748c5d2d-505a-4ac4-8563-d445b5ba48de"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:10:10.028927 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:10:10.028908 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/748c5d2d-505a-4ac4-8563-d445b5ba48de-kube-api-access-72j2b" pod "748c5d2d-505a-4ac4-8563-d445b5ba48de" (UID: "748c5d2d-505a-4ac4-8563-d445b5ba48de"). InnerVolumeSpecName "kube-api-access-72j2b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:10:10.028967 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:10:10.028910 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/748c5d2d-505a-4ac4-8563-d445b5ba48de-istio-podinfo" pod "748c5d2d-505a-4ac4-8563-d445b5ba48de" (UID: "748c5d2d-505a-4ac4-8563-d445b5ba48de"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:10:10.128902 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:10:10.128859 5437 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/748c5d2d-505a-4ac4-8563-d445b5ba48de-istio-envoy\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:10:10.128902 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:10:10.128893 5437 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/748c5d2d-505a-4ac4-8563-d445b5ba48de-istio-token\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:10:10.128902 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:10:10.128906 5437 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/748c5d2d-505a-4ac4-8563-d445b5ba48de-istio-podinfo\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:10:10.129152 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:10:10.128919 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72j2b\" (UniqueName: \"kubernetes.io/projected/748c5d2d-505a-4ac4-8563-d445b5ba48de-kube-api-access-72j2b\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:10:10.176648 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:10:10.176610 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:10:10.348403 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:10:10.348337 5437 generic.go:356] "Generic (PLEG): container finished" podID="748c5d2d-505a-4ac4-8563-d445b5ba48de" containerID="bb1e63d91d134c2b2812ece736cfcce960f30b8524eedf283f6d09fe493edf2b" exitCode=0 Nov 03 02:10:10.348594 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:10:10.348410 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-92zjz" Nov 03 02:10:10.348594 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:10:10.348425 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-92zjz" event={"ID":"748c5d2d-505a-4ac4-8563-d445b5ba48de","Type":"ContainerDied","Data":"bb1e63d91d134c2b2812ece736cfcce960f30b8524eedf283f6d09fe493edf2b"} Nov 03 02:10:10.348594 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:10:10.348502 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-92zjz" event={"ID":"748c5d2d-505a-4ac4-8563-d445b5ba48de","Type":"ContainerDied","Data":"c3da2fd167c9b99a14e9de4bdbd9e116578ba991cf8555cb97ac5759bfbfd98f"} Nov 03 02:10:10.348594 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:10:10.348546 5437 scope.go:122] "RemoveContainer" containerID="bb1e63d91d134c2b2812ece736cfcce960f30b8524eedf283f6d09fe493edf2b" Nov 03 02:10:10.359117 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:10:10.359097 5437 scope.go:122] "RemoveContainer" containerID="bb1e63d91d134c2b2812ece736cfcce960f30b8524eedf283f6d09fe493edf2b" Nov 03 02:10:10.359584 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:10:10.359564 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb1e63d91d134c2b2812ece736cfcce960f30b8524eedf283f6d09fe493edf2b\": container with ID starting with bb1e63d91d134c2b2812ece736cfcce960f30b8524eedf283f6d09fe493edf2b not found: ID does not exist" containerID="bb1e63d91d134c2b2812ece736cfcce960f30b8524eedf283f6d09fe493edf2b" Nov 03 02:10:10.359672 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:10:10.359602 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb1e63d91d134c2b2812ece736cfcce960f30b8524eedf283f6d09fe493edf2b"} err="failed to get container status \"bb1e63d91d134c2b2812ece736cfcce960f30b8524eedf283f6d09fe493edf2b\": rpc error: code = NotFound desc = could not find container \"bb1e63d91d134c2b2812ece736cfcce960f30b8524eedf283f6d09fe493edf2b\": container with ID starting with bb1e63d91d134c2b2812ece736cfcce960f30b8524eedf283f6d09fe493edf2b not found: ID does not exist" Nov 03 02:10:10.427809 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:10:10.427773 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:10:10.519357 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:10:10.519322 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-92zjz"] Nov 03 02:10:10.570988 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:10:10.570949 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-92zjz"] Nov 03 02:10:11.646893 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:10:11.646853 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:10:12.090335 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:10:12.090301 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="748c5d2d-505a-4ac4-8563-d445b5ba48de" path="/var/lib/kubelet/pods/748c5d2d-505a-4ac4-8563-d445b5ba48de/volumes" Nov 03 02:10:14.036789 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:10:14.036750 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 02:10:16.381341 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:10:16.381301 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:10:21.216464 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:10:21.216409 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:10:25.428359 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:10:25.428319 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:10:26.104287 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:10:26.104244 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:10:30.878830 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:10:30.878779 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:10:35.704180 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:10:35.704141 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:10:40.431057 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:10:40.431015 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:10:44.036150 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:10:44.036109 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 02:10:45.215328 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:10:45.215282 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:10:52.057944 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:10:52.057904 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:10:52.262588 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:10:52.262550 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:10:53.688303 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:10:53.688265 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:10:54.372055 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:10:54.372020 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:10:54.783050 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:10:54.782954 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:10:58.379168 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:10:58.379131 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:10:59.498198 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:10:59.498151 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:11:03.647061 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:03.647022 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:11:05.352754 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:05.352713 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:11:05.563350 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:05.563302 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:11:05.930011 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:05.929971 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:11:06.706966 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:06.706930 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:11:10.391940 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:10.391887 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:11:11.444897 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:11.444863 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-b2x7h"] Nov 03 02:11:11.445413 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:11.445148 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-b2x7h" podUID="8d8023ca-6b91-43aa-b753-9ff28068ca28" containerName="istio-proxy" containerID="cri-o://b5d7e6a5d8209bdeeea0e078af7684e0bc0a1e662de5bb7ed153d87356547bea" gracePeriod=30 Nov 03 02:11:12.811094 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:12.811050 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:11:13.272208 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:13.272123 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-f7f61d-6dcb8fbbb5-n2wb4"] Nov 03 02:11:13.370300 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:13.370264 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-f7f61d-7cf44f54d7-xg8mf"] Nov 03 02:11:13.674238 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:13.674196 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:11:13.885480 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:13.885441 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:11:14.036148 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:11:14.036061 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 02:11:14.093587 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:14.093549 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:11:15.413262 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:15.413226 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:11:16.088142 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:16.088108 5437 patch_prober.go:29] interesting pod/e2e-worker-w2-9da431-openshift-default-65c799fd77-b2x7h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 02:11:16.088324 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:16.088175 5437 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-b2x7h" podUID="8d8023ca-6b91-43aa-b753-9ff28068ca28" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 02:11:16.710732 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:16.710702 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-b2x7h" Nov 03 02:11:16.806891 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:16.806855 5437 generic.go:356] "Generic (PLEG): container finished" podID="8d8023ca-6b91-43aa-b753-9ff28068ca28" containerID="b5d7e6a5d8209bdeeea0e078af7684e0bc0a1e662de5bb7ed153d87356547bea" exitCode=0 Nov 03 02:11:16.807106 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:16.806908 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-b2x7h" Nov 03 02:11:16.807106 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:16.806946 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-b2x7h" event={"ID":"8d8023ca-6b91-43aa-b753-9ff28068ca28","Type":"ContainerDied","Data":"b5d7e6a5d8209bdeeea0e078af7684e0bc0a1e662de5bb7ed153d87356547bea"} Nov 03 02:11:16.807106 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:16.807049 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-b2x7h" event={"ID":"8d8023ca-6b91-43aa-b753-9ff28068ca28","Type":"ContainerDied","Data":"04b8acd0872a23d1960d442b86a095706433933ffeb6e30af21173fa2d8137f0"} Nov 03 02:11:16.807106 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:16.807082 5437 scope.go:122] "RemoveContainer" containerID="b5d7e6a5d8209bdeeea0e078af7684e0bc0a1e662de5bb7ed153d87356547bea" Nov 03 02:11:16.817831 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:16.817805 5437 scope.go:122] "RemoveContainer" containerID="b5d7e6a5d8209bdeeea0e078af7684e0bc0a1e662de5bb7ed153d87356547bea" Nov 03 02:11:16.818177 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:11:16.818155 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5d7e6a5d8209bdeeea0e078af7684e0bc0a1e662de5bb7ed153d87356547bea\": container with ID starting with b5d7e6a5d8209bdeeea0e078af7684e0bc0a1e662de5bb7ed153d87356547bea not found: ID does not exist" containerID="b5d7e6a5d8209bdeeea0e078af7684e0bc0a1e662de5bb7ed153d87356547bea" Nov 03 02:11:16.818255 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:16.818186 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5d7e6a5d8209bdeeea0e078af7684e0bc0a1e662de5bb7ed153d87356547bea"} err="failed to get container status \"b5d7e6a5d8209bdeeea0e078af7684e0bc0a1e662de5bb7ed153d87356547bea\": rpc error: code = NotFound desc = could not find container \"b5d7e6a5d8209bdeeea0e078af7684e0bc0a1e662de5bb7ed153d87356547bea\": container with ID starting with b5d7e6a5d8209bdeeea0e078af7684e0bc0a1e662de5bb7ed153d87356547bea not found: ID does not exist" Nov 03 02:11:16.830706 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:16.830632 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d8023ca-6b91-43aa-b753-9ff28068ca28-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d8023ca-6b91-43aa-b753-9ff28068ca28-istio-data\") pod \"8d8023ca-6b91-43aa-b753-9ff28068ca28\" (UID: \"8d8023ca-6b91-43aa-b753-9ff28068ca28\") " Nov 03 02:11:16.830706 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:16.830684 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d8023ca-6b91-43aa-b753-9ff28068ca28-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d8023ca-6b91-43aa-b753-9ff28068ca28-workload-certs\") pod \"8d8023ca-6b91-43aa-b753-9ff28068ca28\" (UID: \"8d8023ca-6b91-43aa-b753-9ff28068ca28\") " Nov 03 02:11:16.830923 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:16.830722 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d8023ca-6b91-43aa-b753-9ff28068ca28-istio-token\" (UniqueName: \"kubernetes.io/projected/8d8023ca-6b91-43aa-b753-9ff28068ca28-istio-token\") pod \"8d8023ca-6b91-43aa-b753-9ff28068ca28\" (UID: \"8d8023ca-6b91-43aa-b753-9ff28068ca28\") " Nov 03 02:11:16.830923 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:16.830785 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d8023ca-6b91-43aa-b753-9ff28068ca28-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d8023ca-6b91-43aa-b753-9ff28068ca28-credential-socket\") pod \"8d8023ca-6b91-43aa-b753-9ff28068ca28\" (UID: \"8d8023ca-6b91-43aa-b753-9ff28068ca28\") " Nov 03 02:11:16.830923 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:16.830856 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d8023ca-6b91-43aa-b753-9ff28068ca28-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d8023ca-6b91-43aa-b753-9ff28068ca28-istio-envoy\") pod \"8d8023ca-6b91-43aa-b753-9ff28068ca28\" (UID: \"8d8023ca-6b91-43aa-b753-9ff28068ca28\") " Nov 03 02:11:16.830923 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:16.830904 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d8023ca-6b91-43aa-b753-9ff28068ca28-istio-data" pod "8d8023ca-6b91-43aa-b753-9ff28068ca28" (UID: "8d8023ca-6b91-43aa-b753-9ff28068ca28"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:11:16.831142 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:16.830954 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d8023ca-6b91-43aa-b753-9ff28068ca28-workload-certs" pod "8d8023ca-6b91-43aa-b753-9ff28068ca28" (UID: "8d8023ca-6b91-43aa-b753-9ff28068ca28"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:11:16.831142 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:16.830975 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d8023ca-6b91-43aa-b753-9ff28068ca28-kube-api-access-v75tj\" (UniqueName: \"kubernetes.io/projected/8d8023ca-6b91-43aa-b753-9ff28068ca28-kube-api-access-v75tj\") pod \"8d8023ca-6b91-43aa-b753-9ff28068ca28\" (UID: \"8d8023ca-6b91-43aa-b753-9ff28068ca28\") " Nov 03 02:11:16.831142 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:16.831057 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8d8023ca-6b91-43aa-b753-9ff28068ca28-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d8023ca-6b91-43aa-b753-9ff28068ca28-istio-podinfo\") pod \"8d8023ca-6b91-43aa-b753-9ff28068ca28\" (UID: \"8d8023ca-6b91-43aa-b753-9ff28068ca28\") " Nov 03 02:11:16.831290 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:16.831167 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d8023ca-6b91-43aa-b753-9ff28068ca28-credential-socket" pod "8d8023ca-6b91-43aa-b753-9ff28068ca28" (UID: "8d8023ca-6b91-43aa-b753-9ff28068ca28"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:11:16.831290 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:16.831181 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d8023ca-6b91-43aa-b753-9ff28068ca28-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d8023ca-6b91-43aa-b753-9ff28068ca28-workload-socket\") pod \"8d8023ca-6b91-43aa-b753-9ff28068ca28\" (UID: \"8d8023ca-6b91-43aa-b753-9ff28068ca28\") " Nov 03 02:11:16.831290 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:16.831238 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d8023ca-6b91-43aa-b753-9ff28068ca28-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d8023ca-6b91-43aa-b753-9ff28068ca28-istiod-ca-cert\") pod \"8d8023ca-6b91-43aa-b753-9ff28068ca28\" (UID: \"8d8023ca-6b91-43aa-b753-9ff28068ca28\") " Nov 03 02:11:16.831509 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:16.831406 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d8023ca-6b91-43aa-b753-9ff28068ca28-workload-socket" pod "8d8023ca-6b91-43aa-b753-9ff28068ca28" (UID: "8d8023ca-6b91-43aa-b753-9ff28068ca28"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:11:16.831719 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:16.831696 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d8023ca-6b91-43aa-b753-9ff28068ca28-istiod-ca-cert" pod "8d8023ca-6b91-43aa-b753-9ff28068ca28" (UID: "8d8023ca-6b91-43aa-b753-9ff28068ca28"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:11:16.831927 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:16.831903 5437 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d8023ca-6b91-43aa-b753-9ff28068ca28-credential-socket\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:11:16.831973 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:16.831936 5437 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d8023ca-6b91-43aa-b753-9ff28068ca28-workload-socket\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:11:16.831973 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:16.831952 5437 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d8023ca-6b91-43aa-b753-9ff28068ca28-istiod-ca-cert\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:11:16.831973 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:16.831968 5437 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d8023ca-6b91-43aa-b753-9ff28068ca28-istio-data\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:11:16.832069 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:16.831983 5437 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d8023ca-6b91-43aa-b753-9ff28068ca28-workload-certs\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:11:16.833169 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:16.833143 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d8023ca-6b91-43aa-b753-9ff28068ca28-istio-envoy" pod "8d8023ca-6b91-43aa-b753-9ff28068ca28" (UID: "8d8023ca-6b91-43aa-b753-9ff28068ca28"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:11:16.833276 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:16.833192 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d8023ca-6b91-43aa-b753-9ff28068ca28-kube-api-access-v75tj" pod "8d8023ca-6b91-43aa-b753-9ff28068ca28" (UID: "8d8023ca-6b91-43aa-b753-9ff28068ca28"). InnerVolumeSpecName "kube-api-access-v75tj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:11:16.833414 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:16.833393 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8d8023ca-6b91-43aa-b753-9ff28068ca28-istio-podinfo" pod "8d8023ca-6b91-43aa-b753-9ff28068ca28" (UID: "8d8023ca-6b91-43aa-b753-9ff28068ca28"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:11:16.833523 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:16.833462 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d8023ca-6b91-43aa-b753-9ff28068ca28-istio-token" pod "8d8023ca-6b91-43aa-b753-9ff28068ca28" (UID: "8d8023ca-6b91-43aa-b753-9ff28068ca28"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:11:16.933762 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:16.933704 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v75tj\" (UniqueName: \"kubernetes.io/projected/8d8023ca-6b91-43aa-b753-9ff28068ca28-kube-api-access-v75tj\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:11:16.933762 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:16.933755 5437 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d8023ca-6b91-43aa-b753-9ff28068ca28-istio-podinfo\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:11:16.933762 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:16.933770 5437 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d8023ca-6b91-43aa-b753-9ff28068ca28-istio-token\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:11:16.934022 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:16.933783 5437 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d8023ca-6b91-43aa-b753-9ff28068ca28-istio-envoy\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:11:17.233615 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:17.233576 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-b2x7h"] Nov 03 02:11:17.241575 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:17.241543 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-b2x7h"] Nov 03 02:11:18.091210 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:18.091170 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d8023ca-6b91-43aa-b753-9ff28068ca28" path="/var/lib/kubelet/pods/8d8023ca-6b91-43aa-b753-9ff28068ca28/volumes" Nov 03 02:11:18.273429 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:18.273346 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-f7f61d-6dcb8fbbb5-n2wb4" podUID="0773a31e-b378-4c6a-9f35-7ba46f2e0dc7" containerName="payload-processing" containerID="cri-o://70acd089881998a1041dca893d73bab0ab3b441d5e59e5243d844e8feb4f6a21" gracePeriod=25 Nov 03 02:11:18.371549 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:18.371418 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f7f61d-7cf44f54d7-xg8mf" podUID="11b45fe9-41b7-445c-b886-34b15a5e0d8a" containerName="payload-pre-processing" containerID="cri-o://a0ba4c7378aa60988f17a7f9a0275822a563103ed4d882e240532e43221f28e1" gracePeriod=25 Nov 03 02:11:18.578462 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:18.578424 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-f7f61d-6dcb8fbbb5-n2wb4" Nov 03 02:11:18.645449 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:18.645422 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f7f61d-7cf44f54d7-xg8mf" Nov 03 02:11:18.653449 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:18.653379 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0773a31e-b378-4c6a-9f35-7ba46f2e0dc7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0773a31e-b378-4c6a-9f35-7ba46f2e0dc7-plugins-config-volume\") pod \"0773a31e-b378-4c6a-9f35-7ba46f2e0dc7\" (UID: \"0773a31e-b378-4c6a-9f35-7ba46f2e0dc7\") " Nov 03 02:11:18.653622 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:18.653502 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11b45fe9-41b7-445c-b886-34b15a5e0d8a-kube-api-access-hskfw\" (UniqueName: \"kubernetes.io/projected/11b45fe9-41b7-445c-b886-34b15a5e0d8a-kube-api-access-hskfw\") pod \"11b45fe9-41b7-445c-b886-34b15a5e0d8a\" (UID: \"11b45fe9-41b7-445c-b886-34b15a5e0d8a\") " Nov 03 02:11:18.653622 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:18.653537 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0773a31e-b378-4c6a-9f35-7ba46f2e0dc7-kube-api-access-mzg2t\" (UniqueName: \"kubernetes.io/projected/0773a31e-b378-4c6a-9f35-7ba46f2e0dc7-kube-api-access-mzg2t\") pod \"0773a31e-b378-4c6a-9f35-7ba46f2e0dc7\" (UID: \"0773a31e-b378-4c6a-9f35-7ba46f2e0dc7\") " Nov 03 02:11:18.653741 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:18.653725 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11b45fe9-41b7-445c-b886-34b15a5e0d8a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11b45fe9-41b7-445c-b886-34b15a5e0d8a-plugins-config-volume\") pod \"11b45fe9-41b7-445c-b886-34b15a5e0d8a\" (UID: \"11b45fe9-41b7-445c-b886-34b15a5e0d8a\") " Nov 03 02:11:18.653809 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:18.653780 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0773a31e-b378-4c6a-9f35-7ba46f2e0dc7-plugins-config-volume" pod "0773a31e-b378-4c6a-9f35-7ba46f2e0dc7" (UID: "0773a31e-b378-4c6a-9f35-7ba46f2e0dc7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:11:18.654079 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:18.654059 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11b45fe9-41b7-445c-b886-34b15a5e0d8a-plugins-config-volume" pod "11b45fe9-41b7-445c-b886-34b15a5e0d8a" (UID: "11b45fe9-41b7-445c-b886-34b15a5e0d8a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:11:18.654157 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:18.654145 5437 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11b45fe9-41b7-445c-b886-34b15a5e0d8a-plugins-config-volume\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:11:18.654203 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:18.654160 5437 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0773a31e-b378-4c6a-9f35-7ba46f2e0dc7-plugins-config-volume\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:11:18.655733 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:18.655707 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11b45fe9-41b7-445c-b886-34b15a5e0d8a-kube-api-access-hskfw" pod "11b45fe9-41b7-445c-b886-34b15a5e0d8a" (UID: "11b45fe9-41b7-445c-b886-34b15a5e0d8a"). InnerVolumeSpecName "kube-api-access-hskfw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:11:18.655831 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:18.655745 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0773a31e-b378-4c6a-9f35-7ba46f2e0dc7-kube-api-access-mzg2t" pod "0773a31e-b378-4c6a-9f35-7ba46f2e0dc7" (UID: "0773a31e-b378-4c6a-9f35-7ba46f2e0dc7"). InnerVolumeSpecName "kube-api-access-mzg2t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:11:18.657175 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:18.657135 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:11:18.755895 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:18.755832 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hskfw\" (UniqueName: \"kubernetes.io/projected/11b45fe9-41b7-445c-b886-34b15a5e0d8a-kube-api-access-hskfw\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:11:18.755895 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:18.755883 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzg2t\" (UniqueName: \"kubernetes.io/projected/0773a31e-b378-4c6a-9f35-7ba46f2e0dc7-kube-api-access-mzg2t\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:11:18.825944 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:18.825906 5437 generic.go:356] "Generic (PLEG): container finished" podID="0773a31e-b378-4c6a-9f35-7ba46f2e0dc7" containerID="70acd089881998a1041dca893d73bab0ab3b441d5e59e5243d844e8feb4f6a21" exitCode=0 Nov 03 02:11:18.826113 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:18.825957 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-f7f61d-6dcb8fbbb5-n2wb4" Nov 03 02:11:18.826113 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:18.825983 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-f7f61d-6dcb8fbbb5-n2wb4" event={"ID":"0773a31e-b378-4c6a-9f35-7ba46f2e0dc7","Type":"ContainerDied","Data":"70acd089881998a1041dca893d73bab0ab3b441d5e59e5243d844e8feb4f6a21"} Nov 03 02:11:18.826113 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:18.826034 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-f7f61d-6dcb8fbbb5-n2wb4" event={"ID":"0773a31e-b378-4c6a-9f35-7ba46f2e0dc7","Type":"ContainerDied","Data":"2c87fa2c18b9ce2003719f8ed9877fa9d11fcff4659a719570b8c5009cb3f9ce"} Nov 03 02:11:18.826113 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:18.826060 5437 scope.go:122] "RemoveContainer" containerID="70acd089881998a1041dca893d73bab0ab3b441d5e59e5243d844e8feb4f6a21" Nov 03 02:11:18.828107 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:18.828082 5437 generic.go:356] "Generic (PLEG): container finished" podID="11b45fe9-41b7-445c-b886-34b15a5e0d8a" containerID="a0ba4c7378aa60988f17a7f9a0275822a563103ed4d882e240532e43221f28e1" exitCode=0 Nov 03 02:11:18.828246 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:18.828123 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f7f61d-7cf44f54d7-xg8mf" Nov 03 02:11:18.828246 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:18.828206 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f7f61d-7cf44f54d7-xg8mf" event={"ID":"11b45fe9-41b7-445c-b886-34b15a5e0d8a","Type":"ContainerDied","Data":"a0ba4c7378aa60988f17a7f9a0275822a563103ed4d882e240532e43221f28e1"} Nov 03 02:11:18.828393 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:18.828244 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f7f61d-7cf44f54d7-xg8mf" event={"ID":"11b45fe9-41b7-445c-b886-34b15a5e0d8a","Type":"ContainerDied","Data":"022ab0168c30aee2efd9e71f58c43cea0c206b870367d6a5c137fd34d9dbf63b"} Nov 03 02:11:18.839818 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:18.839767 5437 scope.go:122] "RemoveContainer" containerID="70acd089881998a1041dca893d73bab0ab3b441d5e59e5243d844e8feb4f6a21" Nov 03 02:11:18.840263 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:11:18.840237 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70acd089881998a1041dca893d73bab0ab3b441d5e59e5243d844e8feb4f6a21\": container with ID starting with 70acd089881998a1041dca893d73bab0ab3b441d5e59e5243d844e8feb4f6a21 not found: ID does not exist" containerID="70acd089881998a1041dca893d73bab0ab3b441d5e59e5243d844e8feb4f6a21" Nov 03 02:11:18.840393 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:18.840273 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70acd089881998a1041dca893d73bab0ab3b441d5e59e5243d844e8feb4f6a21"} err="failed to get container status \"70acd089881998a1041dca893d73bab0ab3b441d5e59e5243d844e8feb4f6a21\": rpc error: code = NotFound desc = could not find container \"70acd089881998a1041dca893d73bab0ab3b441d5e59e5243d844e8feb4f6a21\": container with ID starting with 70acd089881998a1041dca893d73bab0ab3b441d5e59e5243d844e8feb4f6a21 not found: ID does not exist" Nov 03 02:11:18.840393 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:18.840299 5437 scope.go:122] "RemoveContainer" containerID="a0ba4c7378aa60988f17a7f9a0275822a563103ed4d882e240532e43221f28e1" Nov 03 02:11:18.851003 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:18.850967 5437 scope.go:122] "RemoveContainer" containerID="a0ba4c7378aa60988f17a7f9a0275822a563103ed4d882e240532e43221f28e1" Nov 03 02:11:18.851310 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:11:18.851288 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0ba4c7378aa60988f17a7f9a0275822a563103ed4d882e240532e43221f28e1\": container with ID starting with a0ba4c7378aa60988f17a7f9a0275822a563103ed4d882e240532e43221f28e1 not found: ID does not exist" containerID="a0ba4c7378aa60988f17a7f9a0275822a563103ed4d882e240532e43221f28e1" Nov 03 02:11:18.851503 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:18.851319 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0ba4c7378aa60988f17a7f9a0275822a563103ed4d882e240532e43221f28e1"} err="failed to get container status \"a0ba4c7378aa60988f17a7f9a0275822a563103ed4d882e240532e43221f28e1\": rpc error: code = NotFound desc = could not find container \"a0ba4c7378aa60988f17a7f9a0275822a563103ed4d882e240532e43221f28e1\": container with ID starting with a0ba4c7378aa60988f17a7f9a0275822a563103ed4d882e240532e43221f28e1 not found: ID does not exist" Nov 03 02:11:18.997617 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:18.997538 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-f7f61d-7cf44f54d7-xg8mf"] Nov 03 02:11:19.004035 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:19.004004 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-f7f61d-7cf44f54d7-xg8mf"] Nov 03 02:11:19.116677 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:19.116641 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-f7f61d-6dcb8fbbb5-n2wb4"] Nov 03 02:11:19.121759 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:19.121729 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-f7f61d-6dcb8fbbb5-n2wb4"] Nov 03 02:11:20.089200 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:20.089166 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0773a31e-b378-4c6a-9f35-7ba46f2e0dc7" path="/var/lib/kubelet/pods/0773a31e-b378-4c6a-9f35-7ba46f2e0dc7/volumes" Nov 03 02:11:20.089699 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:20.089683 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11b45fe9-41b7-445c-b886-34b15a5e0d8a" path="/var/lib/kubelet/pods/11b45fe9-41b7-445c-b886-34b15a5e0d8a/volumes" Nov 03 02:11:20.235739 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:20.234632 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:11:24.958696 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:24.958648 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:11:29.676179 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:29.676138 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:11:33.833460 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:33.833424 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:11:35.364195 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:35.364155 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-2wlgk"] Nov 03 02:11:35.364739 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:35.364433 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-2wlgk" podUID="84890f73-4e19-4ce2-a5f6-76c9fc295f02" containerName="istio-proxy" containerID="cri-o://343fd6c6588636c1f4a848f3b86966ddfb555f418b1090fe328b3545aff1495a" gracePeriod=30 Nov 03 02:11:35.549402 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:35.549348 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:11:40.303297 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:40.303252 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:11:40.644205 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:40.644174 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-2wlgk" Nov 03 02:11:40.746996 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:40.746948 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84890f73-4e19-4ce2-a5f6-76c9fc295f02-kube-api-access-8kgm5\" (UniqueName: \"kubernetes.io/projected/84890f73-4e19-4ce2-a5f6-76c9fc295f02-kube-api-access-8kgm5\") pod \"84890f73-4e19-4ce2-a5f6-76c9fc295f02\" (UID: \"84890f73-4e19-4ce2-a5f6-76c9fc295f02\") " Nov 03 02:11:40.747218 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:40.747060 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84890f73-4e19-4ce2-a5f6-76c9fc295f02-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84890f73-4e19-4ce2-a5f6-76c9fc295f02-workload-certs\") pod \"84890f73-4e19-4ce2-a5f6-76c9fc295f02\" (UID: \"84890f73-4e19-4ce2-a5f6-76c9fc295f02\") " Nov 03 02:11:40.747218 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:40.747130 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84890f73-4e19-4ce2-a5f6-76c9fc295f02-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84890f73-4e19-4ce2-a5f6-76c9fc295f02-istio-envoy\") pod \"84890f73-4e19-4ce2-a5f6-76c9fc295f02\" (UID: \"84890f73-4e19-4ce2-a5f6-76c9fc295f02\") " Nov 03 02:11:40.747342 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:40.747250 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84890f73-4e19-4ce2-a5f6-76c9fc295f02-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84890f73-4e19-4ce2-a5f6-76c9fc295f02-workload-socket\") pod \"84890f73-4e19-4ce2-a5f6-76c9fc295f02\" (UID: \"84890f73-4e19-4ce2-a5f6-76c9fc295f02\") " Nov 03 02:11:40.747419 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:40.747347 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/84890f73-4e19-4ce2-a5f6-76c9fc295f02-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84890f73-4e19-4ce2-a5f6-76c9fc295f02-istio-podinfo\") pod \"84890f73-4e19-4ce2-a5f6-76c9fc295f02\" (UID: \"84890f73-4e19-4ce2-a5f6-76c9fc295f02\") " Nov 03 02:11:40.747481 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:40.747430 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84890f73-4e19-4ce2-a5f6-76c9fc295f02-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84890f73-4e19-4ce2-a5f6-76c9fc295f02-credential-socket\") pod \"84890f73-4e19-4ce2-a5f6-76c9fc295f02\" (UID: \"84890f73-4e19-4ce2-a5f6-76c9fc295f02\") " Nov 03 02:11:40.747481 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:40.747460 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84890f73-4e19-4ce2-a5f6-76c9fc295f02-workload-certs" pod "84890f73-4e19-4ce2-a5f6-76c9fc295f02" (UID: "84890f73-4e19-4ce2-a5f6-76c9fc295f02"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:11:40.747593 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:40.747470 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84890f73-4e19-4ce2-a5f6-76c9fc295f02-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84890f73-4e19-4ce2-a5f6-76c9fc295f02-istiod-ca-cert\") pod \"84890f73-4e19-4ce2-a5f6-76c9fc295f02\" (UID: \"84890f73-4e19-4ce2-a5f6-76c9fc295f02\") " Nov 03 02:11:40.747593 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:40.747570 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84890f73-4e19-4ce2-a5f6-76c9fc295f02-workload-socket" pod "84890f73-4e19-4ce2-a5f6-76c9fc295f02" (UID: "84890f73-4e19-4ce2-a5f6-76c9fc295f02"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:11:40.747716 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:40.747598 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84890f73-4e19-4ce2-a5f6-76c9fc295f02-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84890f73-4e19-4ce2-a5f6-76c9fc295f02-istio-data\") pod \"84890f73-4e19-4ce2-a5f6-76c9fc295f02\" (UID: \"84890f73-4e19-4ce2-a5f6-76c9fc295f02\") " Nov 03 02:11:40.747716 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:40.747644 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84890f73-4e19-4ce2-a5f6-76c9fc295f02-istio-token\" (UniqueName: \"kubernetes.io/projected/84890f73-4e19-4ce2-a5f6-76c9fc295f02-istio-token\") pod \"84890f73-4e19-4ce2-a5f6-76c9fc295f02\" (UID: \"84890f73-4e19-4ce2-a5f6-76c9fc295f02\") " Nov 03 02:11:40.747825 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:40.747810 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84890f73-4e19-4ce2-a5f6-76c9fc295f02-istiod-ca-cert" pod "84890f73-4e19-4ce2-a5f6-76c9fc295f02" (UID: "84890f73-4e19-4ce2-a5f6-76c9fc295f02"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:11:40.747880 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:40.747826 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84890f73-4e19-4ce2-a5f6-76c9fc295f02-istio-data" pod "84890f73-4e19-4ce2-a5f6-76c9fc295f02" (UID: "84890f73-4e19-4ce2-a5f6-76c9fc295f02"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:11:40.748162 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:40.747739 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84890f73-4e19-4ce2-a5f6-76c9fc295f02-credential-socket" pod "84890f73-4e19-4ce2-a5f6-76c9fc295f02" (UID: "84890f73-4e19-4ce2-a5f6-76c9fc295f02"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:11:40.748299 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:40.748247 5437 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84890f73-4e19-4ce2-a5f6-76c9fc295f02-credential-socket\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:11:40.748299 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:40.748265 5437 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84890f73-4e19-4ce2-a5f6-76c9fc295f02-istiod-ca-cert\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:11:40.748299 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:40.748281 5437 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84890f73-4e19-4ce2-a5f6-76c9fc295f02-istio-data\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:11:40.748299 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:40.748296 5437 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84890f73-4e19-4ce2-a5f6-76c9fc295f02-workload-certs\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:11:40.748638 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:40.748311 5437 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84890f73-4e19-4ce2-a5f6-76c9fc295f02-workload-socket\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:11:40.749722 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:40.749695 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84890f73-4e19-4ce2-a5f6-76c9fc295f02-kube-api-access-8kgm5" pod "84890f73-4e19-4ce2-a5f6-76c9fc295f02" (UID: "84890f73-4e19-4ce2-a5f6-76c9fc295f02"). InnerVolumeSpecName "kube-api-access-8kgm5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:11:40.749895 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:40.749867 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84890f73-4e19-4ce2-a5f6-76c9fc295f02-istio-envoy" pod "84890f73-4e19-4ce2-a5f6-76c9fc295f02" (UID: "84890f73-4e19-4ce2-a5f6-76c9fc295f02"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:11:40.750075 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:40.750052 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84890f73-4e19-4ce2-a5f6-76c9fc295f02-istio-token" pod "84890f73-4e19-4ce2-a5f6-76c9fc295f02" (UID: "84890f73-4e19-4ce2-a5f6-76c9fc295f02"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:11:40.750167 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:40.750143 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/84890f73-4e19-4ce2-a5f6-76c9fc295f02-istio-podinfo" pod "84890f73-4e19-4ce2-a5f6-76c9fc295f02" (UID: "84890f73-4e19-4ce2-a5f6-76c9fc295f02"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:11:40.849781 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:40.849737 5437 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84890f73-4e19-4ce2-a5f6-76c9fc295f02-istio-token\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:11:40.849781 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:40.849778 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8kgm5\" (UniqueName: \"kubernetes.io/projected/84890f73-4e19-4ce2-a5f6-76c9fc295f02-kube-api-access-8kgm5\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:11:40.850009 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:40.849791 5437 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84890f73-4e19-4ce2-a5f6-76c9fc295f02-istio-envoy\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:11:40.850009 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:40.849804 5437 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84890f73-4e19-4ce2-a5f6-76c9fc295f02-istio-podinfo\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:11:40.972933 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:40.972838 5437 generic.go:356] "Generic (PLEG): container finished" podID="84890f73-4e19-4ce2-a5f6-76c9fc295f02" containerID="343fd6c6588636c1f4a848f3b86966ddfb555f418b1090fe328b3545aff1495a" exitCode=0 Nov 03 02:11:40.972933 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:40.972892 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-2wlgk" Nov 03 02:11:40.973139 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:40.972893 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-2wlgk" event={"ID":"84890f73-4e19-4ce2-a5f6-76c9fc295f02","Type":"ContainerDied","Data":"343fd6c6588636c1f4a848f3b86966ddfb555f418b1090fe328b3545aff1495a"} Nov 03 02:11:40.973139 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:40.973014 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-2wlgk" event={"ID":"84890f73-4e19-4ce2-a5f6-76c9fc295f02","Type":"ContainerDied","Data":"2950f37cf65cd148d6b4bb8deaa364287b592a7691bf6dce4fb446a9a61d551c"} Nov 03 02:11:40.973139 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:40.973038 5437 scope.go:122] "RemoveContainer" containerID="343fd6c6588636c1f4a848f3b86966ddfb555f418b1090fe328b3545aff1495a" Nov 03 02:11:40.985019 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:40.984989 5437 scope.go:122] "RemoveContainer" containerID="343fd6c6588636c1f4a848f3b86966ddfb555f418b1090fe328b3545aff1495a" Nov 03 02:11:40.985377 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:11:40.985347 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"343fd6c6588636c1f4a848f3b86966ddfb555f418b1090fe328b3545aff1495a\": container with ID starting with 343fd6c6588636c1f4a848f3b86966ddfb555f418b1090fe328b3545aff1495a not found: ID does not exist" containerID="343fd6c6588636c1f4a848f3b86966ddfb555f418b1090fe328b3545aff1495a" Nov 03 02:11:40.985426 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:40.985388 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"343fd6c6588636c1f4a848f3b86966ddfb555f418b1090fe328b3545aff1495a"} err="failed to get container status \"343fd6c6588636c1f4a848f3b86966ddfb555f418b1090fe328b3545aff1495a\": rpc error: code = NotFound desc = could not find container \"343fd6c6588636c1f4a848f3b86966ddfb555f418b1090fe328b3545aff1495a\": container with ID starting with 343fd6c6588636c1f4a848f3b86966ddfb555f418b1090fe328b3545aff1495a not found: ID does not exist" Nov 03 02:11:41.145512 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:41.145468 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-2wlgk"] Nov 03 02:11:41.149651 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:41.149614 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-2wlgk"] Nov 03 02:11:41.697836 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:41.697800 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:11:41.899886 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:41.899843 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:11:42.090080 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:42.090040 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84890f73-4e19-4ce2-a5f6-76c9fc295f02" path="/var/lib/kubelet/pods/84890f73-4e19-4ce2-a5f6-76c9fc295f02/volumes" Nov 03 02:11:43.175320 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:43.175280 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:11:43.729056 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:43.729013 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:11:44.036228 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:11:44.036133 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 02:11:45.249690 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:45.249641 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:11:50.213445 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:50.213403 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:11:53.918780 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:53.918736 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:11:54.944647 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:54.944597 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:11:57.279244 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:57.279203 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:11:58.727176 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:58.727130 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:11:59.685622 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:11:59.685581 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:12:03.632433 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:12:03.632394 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:12:04.256301 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:12:04.256254 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:12:04.516927 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:12:04.516819 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:12:06.817055 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:12:06.817015 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:12:08.268274 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:12:08.268233 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:12:08.486245 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:12:08.486206 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:12:09.387553 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:12:09.387497 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:12:12.447501 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:12:12.447453 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:12:12.655206 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:12:12.655171 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:12:14.036147 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:12:14.036108 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 02:12:14.472011 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:12:14.471971 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:12:19.258897 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:12:19.258859 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:12:24.015155 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:12:24.015117 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:12:28.730414 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:12:28.730347 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:12:33.483567 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:12:33.483509 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:12:38.271277 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:12:38.271236 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:12:43.008864 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:12:43.008816 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:12:44.035797 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:12:44.035759 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 02:12:47.759442 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:12:47.759326 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:12:52.589399 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:12:52.589326 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:12:54.280220 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:12:54.280180 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:12:54.539339 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:12:54.539239 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:12:57.385204 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:12:57.385166 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:12:59.062298 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:12:59.062250 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:13:02.183288 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:13:02.183245 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:13:06.918856 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:13:06.918802 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:13:11.788163 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:13:11.788095 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:13:14.035805 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:13:14.035758 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 02:13:16.439800 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:13:16.439755 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:13:16.691458 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:13:16.691345 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:13:16.896211 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:13:16.896170 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:13:19.559549 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:13:19.559492 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:13:19.768033 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:13:19.767996 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:13:21.355569 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:13:21.355530 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:13:26.186268 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:13:26.186225 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:13:30.962533 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:13:30.962484 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:13:35.681817 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:13:35.681759 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:13:40.416597 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:13:40.416548 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:13:44.035661 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:13:44.035620 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 02:13:45.190950 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:13:45.190911 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:13:49.994943 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:13:49.994904 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:13:54.630622 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:13:54.630580 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:13:54.882145 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:13:54.882056 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:13:59.401788 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:13:59.401748 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:13:59.606703 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:13:59.606664 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:14:04.379012 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:04.378971 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:14:09.173415 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:09.173374 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:14:13.911104 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:13.911067 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:14:14.034654 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:14:14.034620 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 02:14:18.688957 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:18.688869 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:14:21.099180 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:21.099144 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-059b23/auth-test-model-059b23-kserve-797887d945-j9d4h"] Nov 03 02:14:21.155656 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:21.155619 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:14:22.610651 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:22.610595 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:14:22.787815 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:22.787780 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7011-877eae"] Nov 03 02:14:22.788384 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:22.788356 5437 cpu_manager.go:414] "containerMap: removing container" podUID="748c5d2d-505a-4ac4-8563-d445b5ba48de" containerName="istio-proxy" Nov 03 02:14:22.788436 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:22.788387 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="748c5d2d-505a-4ac4-8563-d445b5ba48de" containerName="istio-proxy" Nov 03 02:14:22.788436 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:22.788406 5437 cpu_manager.go:414] "containerMap: removing container" podUID="84890f73-4e19-4ce2-a5f6-76c9fc295f02" containerName="istio-proxy" Nov 03 02:14:22.788436 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:22.788412 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84890f73-4e19-4ce2-a5f6-76c9fc295f02" containerName="istio-proxy" Nov 03 02:14:22.788534 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:22.788439 5437 cpu_manager.go:414] "containerMap: removing container" podUID="11b45fe9-41b7-445c-b886-34b15a5e0d8a" containerName="payload-pre-processing" Nov 03 02:14:22.788534 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:22.788444 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11b45fe9-41b7-445c-b886-34b15a5e0d8a" containerName="payload-pre-processing" Nov 03 02:14:22.788534 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:22.788463 5437 cpu_manager.go:414] "containerMap: removing container" podUID="0773a31e-b378-4c6a-9f35-7ba46f2e0dc7" containerName="payload-processing" Nov 03 02:14:22.788534 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:22.788469 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0773a31e-b378-4c6a-9f35-7ba46f2e0dc7" containerName="payload-processing" Nov 03 02:14:22.788534 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:22.788480 5437 cpu_manager.go:414] "containerMap: removing container" podUID="8d8023ca-6b91-43aa-b753-9ff28068ca28" containerName="istio-proxy" Nov 03 02:14:22.788534 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:22.788484 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d8023ca-6b91-43aa-b753-9ff28068ca28" containerName="istio-proxy" Nov 03 02:14:22.788721 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:22.788585 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="11b45fe9-41b7-445c-b886-34b15a5e0d8a" containerName="payload-pre-processing" Nov 03 02:14:22.788721 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:22.788597 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="0773a31e-b378-4c6a-9f35-7ba46f2e0dc7" containerName="payload-processing" Nov 03 02:14:22.788721 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:22.788604 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d8023ca-6b91-43aa-b753-9ff28068ca28" containerName="istio-proxy" Nov 03 02:14:22.788721 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:22.788613 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="748c5d2d-505a-4ac4-8563-d445b5ba48de" containerName="istio-proxy" Nov 03 02:14:22.788721 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:22.788618 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="84890f73-4e19-4ce2-a5f6-76c9fc295f02" containerName="istio-proxy" Nov 03 02:14:22.792022 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:22.792005 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7011-877eae" Nov 03 02:14:22.843865 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:22.843836 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7011-877eae"] Nov 03 02:14:22.884696 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:22.884600 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxrg2\" (UniqueName: \"kubernetes.io/projected/621f3117-07aa-49e8-bf07-2f4539b87abe-kube-api-access-zxrg2\") pod \"test-curl-7011-877eae\" (UID: \"621f3117-07aa-49e8-bf07-2f4539b87abe\") " pod="opendatahub/test-curl-7011-877eae" Nov 03 02:14:22.892569 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:22.892548 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-882bv\"" Nov 03 02:14:22.985792 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:22.985751 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxrg2\" (UniqueName: \"kubernetes.io/projected/621f3117-07aa-49e8-bf07-2f4539b87abe-kube-api-access-zxrg2\") pod \"test-curl-7011-877eae\" (UID: \"621f3117-07aa-49e8-bf07-2f4539b87abe\") " pod="opendatahub/test-curl-7011-877eae" Nov 03 02:14:23.048916 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:23.048872 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxrg2\" (UniqueName: \"kubernetes.io/projected/621f3117-07aa-49e8-bf07-2f4539b87abe-kube-api-access-zxrg2\") pod \"test-curl-7011-877eae\" (UID: \"621f3117-07aa-49e8-bf07-2f4539b87abe\") " pod="opendatahub/test-curl-7011-877eae" Nov 03 02:14:23.104139 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:23.104107 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7011-877eae" Nov 03 02:14:23.474251 ip-10-0-136-16 kubenswrapper[5437]: W1103 02:14:23.474210 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod621f3117_07aa_49e8_bf07_2f4539b87abe.slice/crio-8e5ef87b4bf03d6c0545a146d409ff9548e7f250e374f0dec15365e08fb25920 WatchSource:0}: Error finding container 8e5ef87b4bf03d6c0545a146d409ff9548e7f250e374f0dec15365e08fb25920: Status 404 returned error can't find the container with id 8e5ef87b4bf03d6c0545a146d409ff9548e7f250e374f0dec15365e08fb25920 Nov 03 02:14:23.476430 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:23.476410 5437 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 02:14:23.480467 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:23.478191 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7011-877eae"] Nov 03 02:14:23.503881 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:23.503843 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:14:24.028455 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:24.028404 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7011-877eae" event={"ID":"621f3117-07aa-49e8-bf07-2f4539b87abe","Type":"ContainerStarted","Data":"8e5ef87b4bf03d6c0545a146d409ff9548e7f250e374f0dec15365e08fb25920"} Nov 03 02:14:24.406019 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:24.405978 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:14:24.608471 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:24.608429 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:14:27.053080 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:27.053035 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7011-877eae" event={"ID":"621f3117-07aa-49e8-bf07-2f4539b87abe","Type":"ContainerStarted","Data":"695d53d2a3a3c474cd7b6f7e9273909e12174e7d0ab703f8746c5a910140d8d6"} Nov 03 02:14:27.172725 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:27.172670 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7011-877eae" podStartSLOduration=2.221464629 podStartE2EDuration="5.172656806s" totalImagesPullingTime="2.951192177s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:14:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:14:27.171001169 +0000 UTC m=+3349.200483464" watchObservedRunningTime="2026-11-03 02:14:27.172656806 +0000 UTC m=+3349.202139100" Nov 03 02:14:27.918565 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:27.918531 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-ngbgz"] Nov 03 02:14:27.918780 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:27.918741 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-ngbgz" podUID="b2740d82-3082-474a-8726-4aaaecf57272" containerName="istio-proxy" containerID="cri-o://875333aad582145899d9f30bb4e88f15e8d54be9541e1f768dc8ebd1a69ff108" gracePeriod=30 Nov 03 02:14:28.285744 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:28.285654 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:14:28.768056 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:28.768024 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7011-877eae"] Nov 03 02:14:28.772021 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:28.771992 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7011-877eae"] Nov 03 02:14:29.064534 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:29.064495 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7011-877eae" podUID="621f3117-07aa-49e8-bf07-2f4539b87abe" containerName="test-curl-7011-877eae" containerID="cri-o://695d53d2a3a3c474cd7b6f7e9273909e12174e7d0ab703f8746c5a910140d8d6" gracePeriod=2 Nov 03 02:14:29.117655 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:14:29.117613 5437 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7011-877eae\" is forbidden: User \"system:node:ip-10-0-136-16.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-16.us-west-2.compute.internal' and this object" podUID="621f3117-07aa-49e8-bf07-2f4539b87abe" pod="opendatahub/test-curl-7011-877eae" Nov 03 02:14:29.140942 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:29.140910 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7011-7ece81"] Nov 03 02:14:29.141558 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:29.141543 5437 cpu_manager.go:414] "containerMap: removing container" podUID="621f3117-07aa-49e8-bf07-2f4539b87abe" containerName="test-curl-7011-877eae" Nov 03 02:14:29.141602 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:29.141559 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="621f3117-07aa-49e8-bf07-2f4539b87abe" containerName="test-curl-7011-877eae" Nov 03 02:14:29.141656 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:29.141647 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="621f3117-07aa-49e8-bf07-2f4539b87abe" containerName="test-curl-7011-877eae" Nov 03 02:14:29.145226 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:29.145209 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7011-7ece81" Nov 03 02:14:29.196300 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:29.196262 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7011-7ece81"] Nov 03 02:14:29.270062 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:29.270024 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27wmb\" (UniqueName: \"kubernetes.io/projected/19e1859b-a75a-4d37-af5c-e6e0bce0921b-kube-api-access-27wmb\") pod \"test-curl-7011-7ece81\" (UID: \"19e1859b-a75a-4d37-af5c-e6e0bce0921b\") " pod="opendatahub/test-curl-7011-7ece81" Nov 03 02:14:29.316270 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:14:29.316188 5437 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7011-877eae\" is forbidden: User \"system:node:ip-10-0-136-16.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-16.us-west-2.compute.internal' and this object" podUID="621f3117-07aa-49e8-bf07-2f4539b87abe" pod="opendatahub/test-curl-7011-877eae" Nov 03 02:14:29.370956 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:29.370922 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27wmb\" (UniqueName: \"kubernetes.io/projected/19e1859b-a75a-4d37-af5c-e6e0bce0921b-kube-api-access-27wmb\") pod \"test-curl-7011-7ece81\" (UID: \"19e1859b-a75a-4d37-af5c-e6e0bce0921b\") " pod="opendatahub/test-curl-7011-7ece81" Nov 03 02:14:29.433678 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:29.433642 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27wmb\" (UniqueName: \"kubernetes.io/projected/19e1859b-a75a-4d37-af5c-e6e0bce0921b-kube-api-access-27wmb\") pod \"test-curl-7011-7ece81\" (UID: \"19e1859b-a75a-4d37-af5c-e6e0bce0921b\") " pod="opendatahub/test-curl-7011-7ece81" Nov 03 02:14:29.457730 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:29.457696 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7011-7ece81" Nov 03 02:14:29.827011 ip-10-0-136-16 kubenswrapper[5437]: W1103 02:14:29.826972 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod19e1859b_a75a_4d37_af5c_e6e0bce0921b.slice/crio-5e38387e18961e5b293e839adfa976324d682e98877b29d26f3eae40223acfed WatchSource:0}: Error finding container 5e38387e18961e5b293e839adfa976324d682e98877b29d26f3eae40223acfed: Status 404 returned error can't find the container with id 5e38387e18961e5b293e839adfa976324d682e98877b29d26f3eae40223acfed Nov 03 02:14:29.830572 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:29.830549 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7011-7ece81"] Nov 03 02:14:30.071997 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:30.071960 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7011-7ece81" event={"ID":"19e1859b-a75a-4d37-af5c-e6e0bce0921b","Type":"ContainerStarted","Data":"b09e27df9716bf743171cd94707803ed8eed14537766965d8fbe7dbf8a993570"} Nov 03 02:14:30.072209 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:30.072008 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7011-7ece81" event={"ID":"19e1859b-a75a-4d37-af5c-e6e0bce0921b","Type":"ContainerStarted","Data":"5e38387e18961e5b293e839adfa976324d682e98877b29d26f3eae40223acfed"} Nov 03 02:14:30.189783 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:30.189666 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7011-7ece81" podStartSLOduration=1.189650845 podStartE2EDuration="1.189650845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:14:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:14:30.188270843 +0000 UTC m=+3352.217753138" watchObservedRunningTime="2026-11-03 02:14:30.189650845 +0000 UTC m=+3352.219133137" Nov 03 02:14:30.241147 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:14:30.241106 5437 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7011-877eae\" is forbidden: User \"system:node:ip-10-0-136-16.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-16.us-west-2.compute.internal' and this object" podUID="621f3117-07aa-49e8-bf07-2f4539b87abe" pod="opendatahub/test-curl-7011-877eae" Nov 03 02:14:31.310733 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:31.310700 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7011-877eae" Nov 03 02:14:31.395947 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:31.395855 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/621f3117-07aa-49e8-bf07-2f4539b87abe-kube-api-access-zxrg2\" (UniqueName: \"kubernetes.io/projected/621f3117-07aa-49e8-bf07-2f4539b87abe-kube-api-access-zxrg2\") pod \"621f3117-07aa-49e8-bf07-2f4539b87abe\" (UID: \"621f3117-07aa-49e8-bf07-2f4539b87abe\") " Nov 03 02:14:31.398097 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:31.398069 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/621f3117-07aa-49e8-bf07-2f4539b87abe-kube-api-access-zxrg2" pod "621f3117-07aa-49e8-bf07-2f4539b87abe" (UID: "621f3117-07aa-49e8-bf07-2f4539b87abe"). InnerVolumeSpecName "kube-api-access-zxrg2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:14:31.497722 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:31.497689 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxrg2\" (UniqueName: \"kubernetes.io/projected/621f3117-07aa-49e8-bf07-2f4539b87abe-kube-api-access-zxrg2\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:14:31.635819 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:31.635775 5437 patch_prober.go:29] interesting pod/e2e-worker-w2-9da431-openshift-default-65c799fd77-ngbgz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 02:14:31.635984 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:31.635839 5437 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-ngbgz" podUID="b2740d82-3082-474a-8726-4aaaecf57272" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 02:14:31.662460 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:31.662382 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7011-7ece81"] Nov 03 02:14:31.666640 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:31.666612 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7011-7ece81"] Nov 03 02:14:31.816032 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:31.815994 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7011-4c2838"] Nov 03 02:14:31.816674 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:31.816657 5437 cpu_manager.go:414] "containerMap: removing container" podUID="19e1859b-a75a-4d37-af5c-e6e0bce0921b" containerName="test-curl-7011-7ece81" Nov 03 02:14:31.816734 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:31.816676 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19e1859b-a75a-4d37-af5c-e6e0bce0921b" containerName="test-curl-7011-7ece81" Nov 03 02:14:31.816888 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:31.816874 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="19e1859b-a75a-4d37-af5c-e6e0bce0921b" containerName="test-curl-7011-7ece81" Nov 03 02:14:31.820505 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:31.820484 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7011-4c2838" Nov 03 02:14:31.870374 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:31.870327 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7011-4c2838"] Nov 03 02:14:31.902438 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:31.902406 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwl2j\" (UniqueName: \"kubernetes.io/projected/d3ceb6c5-67db-4e65-8850-7daee812cf22-kube-api-access-jwl2j\") pod \"test-curl-7011-4c2838\" (UID: \"d3ceb6c5-67db-4e65-8850-7daee812cf22\") " pod="opendatahub/test-curl-7011-4c2838" Nov 03 02:14:32.004327 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:32.004234 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwl2j\" (UniqueName: \"kubernetes.io/projected/d3ceb6c5-67db-4e65-8850-7daee812cf22-kube-api-access-jwl2j\") pod \"test-curl-7011-4c2838\" (UID: \"d3ceb6c5-67db-4e65-8850-7daee812cf22\") " pod="opendatahub/test-curl-7011-4c2838" Nov 03 02:14:32.067137 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:32.067101 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwl2j\" (UniqueName: \"kubernetes.io/projected/d3ceb6c5-67db-4e65-8850-7daee812cf22-kube-api-access-jwl2j\") pod \"test-curl-7011-4c2838\" (UID: \"d3ceb6c5-67db-4e65-8850-7daee812cf22\") " pod="opendatahub/test-curl-7011-4c2838" Nov 03 02:14:32.087290 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:32.087255 5437 generic.go:356] "Generic (PLEG): container finished" podID="621f3117-07aa-49e8-bf07-2f4539b87abe" containerID="695d53d2a3a3c474cd7b6f7e9273909e12174e7d0ab703f8746c5a910140d8d6" exitCode=137 Nov 03 02:14:32.087456 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:32.087390 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7011-877eae" Nov 03 02:14:32.087708 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:32.087665 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7011-7ece81" podUID="19e1859b-a75a-4d37-af5c-e6e0bce0921b" containerName="test-curl-7011-7ece81" containerID="cri-o://b09e27df9716bf743171cd94707803ed8eed14537766965d8fbe7dbf8a993570" gracePeriod=2 Nov 03 02:14:32.090233 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:32.090210 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="621f3117-07aa-49e8-bf07-2f4539b87abe" path="/var/lib/kubelet/pods/621f3117-07aa-49e8-bf07-2f4539b87abe/volumes" Nov 03 02:14:32.090700 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:32.090685 5437 scope.go:122] "RemoveContainer" containerID="695d53d2a3a3c474cd7b6f7e9273909e12174e7d0ab703f8746c5a910140d8d6" Nov 03 02:14:32.101588 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:32.101569 5437 scope.go:122] "RemoveContainer" containerID="695d53d2a3a3c474cd7b6f7e9273909e12174e7d0ab703f8746c5a910140d8d6" Nov 03 02:14:32.101865 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:14:32.101845 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"695d53d2a3a3c474cd7b6f7e9273909e12174e7d0ab703f8746c5a910140d8d6\": container with ID starting with 695d53d2a3a3c474cd7b6f7e9273909e12174e7d0ab703f8746c5a910140d8d6 not found: ID does not exist" containerID="695d53d2a3a3c474cd7b6f7e9273909e12174e7d0ab703f8746c5a910140d8d6" Nov 03 02:14:32.101921 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:32.101872 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"695d53d2a3a3c474cd7b6f7e9273909e12174e7d0ab703f8746c5a910140d8d6"} err="failed to get container status \"695d53d2a3a3c474cd7b6f7e9273909e12174e7d0ab703f8746c5a910140d8d6\": rpc error: code = NotFound desc = could not find container \"695d53d2a3a3c474cd7b6f7e9273909e12174e7d0ab703f8746c5a910140d8d6\": container with ID starting with 695d53d2a3a3c474cd7b6f7e9273909e12174e7d0ab703f8746c5a910140d8d6 not found: ID does not exist" Nov 03 02:14:32.132730 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:32.132703 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7011-4c2838" Nov 03 02:14:32.140702 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:14:32.140673 5437 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7011-7ece81\" is forbidden: User \"system:node:ip-10-0-136-16.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-16.us-west-2.compute.internal' and this object" podUID="19e1859b-a75a-4d37-af5c-e6e0bce0921b" pod="opendatahub/test-curl-7011-7ece81" Nov 03 02:14:32.501252 ip-10-0-136-16 kubenswrapper[5437]: W1103 02:14:32.501210 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd3ceb6c5_67db_4e65_8850_7daee812cf22.slice/crio-0330f3adf2d5b37f7fa0f07aaa5dfbb70d2cd20c63a3e8cae29f422cc6f29579 WatchSource:0}: Error finding container 0330f3adf2d5b37f7fa0f07aaa5dfbb70d2cd20c63a3e8cae29f422cc6f29579: Status 404 returned error can't find the container with id 0330f3adf2d5b37f7fa0f07aaa5dfbb70d2cd20c63a3e8cae29f422cc6f29579 Nov 03 02:14:32.504896 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:32.504869 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7011-4c2838"] Nov 03 02:14:33.080982 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:33.080947 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:14:33.097001 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:33.096970 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7011-4c2838" event={"ID":"d3ceb6c5-67db-4e65-8850-7daee812cf22","Type":"ContainerStarted","Data":"b70d9040756b856c50a0c7230872a5f3ebdd0041fe150e0fd5c3ead74fd6a450"} Nov 03 02:14:33.097124 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:33.097021 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7011-4c2838" event={"ID":"d3ceb6c5-67db-4e65-8850-7daee812cf22","Type":"ContainerStarted","Data":"0330f3adf2d5b37f7fa0f07aaa5dfbb70d2cd20c63a3e8cae29f422cc6f29579"} Nov 03 02:14:33.099136 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:33.099102 5437 generic.go:356] "Generic (PLEG): container finished" podID="b2740d82-3082-474a-8726-4aaaecf57272" containerID="875333aad582145899d9f30bb4e88f15e8d54be9541e1f768dc8ebd1a69ff108" exitCode=0 Nov 03 02:14:33.099277 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:33.099258 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-ngbgz" event={"ID":"b2740d82-3082-474a-8726-4aaaecf57272","Type":"ContainerDied","Data":"875333aad582145899d9f30bb4e88f15e8d54be9541e1f768dc8ebd1a69ff108"} Nov 03 02:14:33.186807 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:33.186742 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-ngbgz" Nov 03 02:14:33.217145 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:33.217092 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7011-4c2838" podStartSLOduration=2.217078492 podStartE2EDuration="2.217078492s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:14:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:14:33.214728809 +0000 UTC m=+3355.244211104" watchObservedRunningTime="2026-11-03 02:14:33.217078492 +0000 UTC m=+3355.246560787" Nov 03 02:14:33.322203 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:33.322169 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2740d82-3082-474a-8726-4aaaecf57272-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2740d82-3082-474a-8726-4aaaecf57272-istiod-ca-cert\") pod \"b2740d82-3082-474a-8726-4aaaecf57272\" (UID: \"b2740d82-3082-474a-8726-4aaaecf57272\") " Nov 03 02:14:33.322410 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:33.322220 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2740d82-3082-474a-8726-4aaaecf57272-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2740d82-3082-474a-8726-4aaaecf57272-workload-certs\") pod \"b2740d82-3082-474a-8726-4aaaecf57272\" (UID: \"b2740d82-3082-474a-8726-4aaaecf57272\") " Nov 03 02:14:33.322410 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:33.322253 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b2740d82-3082-474a-8726-4aaaecf57272-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2740d82-3082-474a-8726-4aaaecf57272-istio-podinfo\") pod \"b2740d82-3082-474a-8726-4aaaecf57272\" (UID: \"b2740d82-3082-474a-8726-4aaaecf57272\") " Nov 03 02:14:33.322410 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:33.322305 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2740d82-3082-474a-8726-4aaaecf57272-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2740d82-3082-474a-8726-4aaaecf57272-workload-socket\") pod \"b2740d82-3082-474a-8726-4aaaecf57272\" (UID: \"b2740d82-3082-474a-8726-4aaaecf57272\") " Nov 03 02:14:33.322410 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:33.322394 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2740d82-3082-474a-8726-4aaaecf57272-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2740d82-3082-474a-8726-4aaaecf57272-istio-data\") pod \"b2740d82-3082-474a-8726-4aaaecf57272\" (UID: \"b2740d82-3082-474a-8726-4aaaecf57272\") " Nov 03 02:14:33.322589 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:33.322451 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2740d82-3082-474a-8726-4aaaecf57272-kube-api-access-krz6h\" (UniqueName: \"kubernetes.io/projected/b2740d82-3082-474a-8726-4aaaecf57272-kube-api-access-krz6h\") pod \"b2740d82-3082-474a-8726-4aaaecf57272\" (UID: \"b2740d82-3082-474a-8726-4aaaecf57272\") " Nov 03 02:14:33.322589 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:33.322530 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2740d82-3082-474a-8726-4aaaecf57272-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2740d82-3082-474a-8726-4aaaecf57272-istio-envoy\") pod \"b2740d82-3082-474a-8726-4aaaecf57272\" (UID: \"b2740d82-3082-474a-8726-4aaaecf57272\") " Nov 03 02:14:33.322589 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:33.322555 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2740d82-3082-474a-8726-4aaaecf57272-istio-token\" (UniqueName: \"kubernetes.io/projected/b2740d82-3082-474a-8726-4aaaecf57272-istio-token\") pod \"b2740d82-3082-474a-8726-4aaaecf57272\" (UID: \"b2740d82-3082-474a-8726-4aaaecf57272\") " Nov 03 02:14:33.322754 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:33.322596 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2740d82-3082-474a-8726-4aaaecf57272-workload-certs" pod "b2740d82-3082-474a-8726-4aaaecf57272" (UID: "b2740d82-3082-474a-8726-4aaaecf57272"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:14:33.322754 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:33.322654 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2740d82-3082-474a-8726-4aaaecf57272-workload-socket" pod "b2740d82-3082-474a-8726-4aaaecf57272" (UID: "b2740d82-3082-474a-8726-4aaaecf57272"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:14:33.322754 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:33.322717 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2740d82-3082-474a-8726-4aaaecf57272-istio-data" pod "b2740d82-3082-474a-8726-4aaaecf57272" (UID: "b2740d82-3082-474a-8726-4aaaecf57272"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:14:33.322754 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:33.322727 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2740d82-3082-474a-8726-4aaaecf57272-istiod-ca-cert" pod "b2740d82-3082-474a-8726-4aaaecf57272" (UID: "b2740d82-3082-474a-8726-4aaaecf57272"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:14:33.322978 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:33.322763 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2740d82-3082-474a-8726-4aaaecf57272-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2740d82-3082-474a-8726-4aaaecf57272-credential-socket\") pod \"b2740d82-3082-474a-8726-4aaaecf57272\" (UID: \"b2740d82-3082-474a-8726-4aaaecf57272\") " Nov 03 02:14:33.323047 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:33.322991 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2740d82-3082-474a-8726-4aaaecf57272-credential-socket" pod "b2740d82-3082-474a-8726-4aaaecf57272" (UID: "b2740d82-3082-474a-8726-4aaaecf57272"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:14:33.323439 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:33.323418 5437 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2740d82-3082-474a-8726-4aaaecf57272-workload-socket\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:14:33.323506 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:33.323444 5437 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2740d82-3082-474a-8726-4aaaecf57272-istio-data\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:14:33.323506 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:33.323460 5437 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2740d82-3082-474a-8726-4aaaecf57272-credential-socket\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:14:33.323506 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:33.323476 5437 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2740d82-3082-474a-8726-4aaaecf57272-istiod-ca-cert\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:14:33.323506 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:33.323490 5437 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2740d82-3082-474a-8726-4aaaecf57272-workload-certs\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:14:33.324779 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:33.324751 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b2740d82-3082-474a-8726-4aaaecf57272-istio-podinfo" pod "b2740d82-3082-474a-8726-4aaaecf57272" (UID: "b2740d82-3082-474a-8726-4aaaecf57272"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:14:33.324877 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:33.324780 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2740d82-3082-474a-8726-4aaaecf57272-kube-api-access-krz6h" pod "b2740d82-3082-474a-8726-4aaaecf57272" (UID: "b2740d82-3082-474a-8726-4aaaecf57272"). InnerVolumeSpecName "kube-api-access-krz6h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:14:33.325091 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:33.325065 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2740d82-3082-474a-8726-4aaaecf57272-istio-envoy" pod "b2740d82-3082-474a-8726-4aaaecf57272" (UID: "b2740d82-3082-474a-8726-4aaaecf57272"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:14:33.325199 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:33.325108 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2740d82-3082-474a-8726-4aaaecf57272-istio-token" pod "b2740d82-3082-474a-8726-4aaaecf57272" (UID: "b2740d82-3082-474a-8726-4aaaecf57272"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:14:33.424871 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:33.424834 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krz6h\" (UniqueName: \"kubernetes.io/projected/b2740d82-3082-474a-8726-4aaaecf57272-kube-api-access-krz6h\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:14:33.424871 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:33.424870 5437 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2740d82-3082-474a-8726-4aaaecf57272-istio-envoy\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:14:33.425086 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:33.424886 5437 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2740d82-3082-474a-8726-4aaaecf57272-istio-token\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:14:33.425086 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:33.424901 5437 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2740d82-3082-474a-8726-4aaaecf57272-istio-podinfo\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:14:34.107059 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:34.107023 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-ngbgz" Nov 03 02:14:34.107432 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:34.107045 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-ngbgz" event={"ID":"b2740d82-3082-474a-8726-4aaaecf57272","Type":"ContainerDied","Data":"622b2b8f64364e1d2231e53585a67424315a4e8d40e424ed75ad37996b6bdc66"} Nov 03 02:14:34.107432 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:34.107113 5437 scope.go:122] "RemoveContainer" containerID="875333aad582145899d9f30bb4e88f15e8d54be9541e1f768dc8ebd1a69ff108" Nov 03 02:14:34.283689 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:34.283652 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-ngbgz"] Nov 03 02:14:34.334938 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:34.334903 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-ngbgz"] Nov 03 02:14:34.341786 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:34.341721 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7011-7ece81" Nov 03 02:14:34.436100 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:34.436068 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19e1859b-a75a-4d37-af5c-e6e0bce0921b-kube-api-access-27wmb\" (UniqueName: \"kubernetes.io/projected/19e1859b-a75a-4d37-af5c-e6e0bce0921b-kube-api-access-27wmb\") pod \"19e1859b-a75a-4d37-af5c-e6e0bce0921b\" (UID: \"19e1859b-a75a-4d37-af5c-e6e0bce0921b\") " Nov 03 02:14:34.438106 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:34.438080 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19e1859b-a75a-4d37-af5c-e6e0bce0921b-kube-api-access-27wmb" pod "19e1859b-a75a-4d37-af5c-e6e0bce0921b" (UID: "19e1859b-a75a-4d37-af5c-e6e0bce0921b"). InnerVolumeSpecName "kube-api-access-27wmb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:14:34.537801 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:34.537768 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27wmb\" (UniqueName: \"kubernetes.io/projected/19e1859b-a75a-4d37-af5c-e6e0bce0921b-kube-api-access-27wmb\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:14:34.740771 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:34.740679 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7011-4c2838"] Nov 03 02:14:34.743111 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:34.743080 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7011-4c2838"] Nov 03 02:14:34.922644 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:34.922609 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7011-700993"] Nov 03 02:14:34.923193 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:34.923180 5437 cpu_manager.go:414] "containerMap: removing container" podUID="d3ceb6c5-67db-4e65-8850-7daee812cf22" containerName="test-curl-7011-4c2838" Nov 03 02:14:34.923239 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:34.923194 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3ceb6c5-67db-4e65-8850-7daee812cf22" containerName="test-curl-7011-4c2838" Nov 03 02:14:34.923239 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:34.923230 5437 cpu_manager.go:414] "containerMap: removing container" podUID="b2740d82-3082-474a-8726-4aaaecf57272" containerName="istio-proxy" Nov 03 02:14:34.923239 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:34.923235 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2740d82-3082-474a-8726-4aaaecf57272" containerName="istio-proxy" Nov 03 02:14:34.923336 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:34.923317 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2740d82-3082-474a-8726-4aaaecf57272" containerName="istio-proxy" Nov 03 02:14:34.923336 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:34.923333 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3ceb6c5-67db-4e65-8850-7daee812cf22" containerName="test-curl-7011-4c2838" Nov 03 02:14:34.928618 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:34.928586 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7011-700993" Nov 03 02:14:34.977133 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:34.977092 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7011-700993"] Nov 03 02:14:35.044682 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:35.044583 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95rws\" (UniqueName: \"kubernetes.io/projected/0346b8eb-e18f-4689-a880-71954310ace0-kube-api-access-95rws\") pod \"test-curl-7011-700993\" (UID: \"0346b8eb-e18f-4689-a880-71954310ace0\") " pod="opendatahub/test-curl-7011-700993" Nov 03 02:14:35.116032 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:35.115993 5437 generic.go:356] "Generic (PLEG): container finished" podID="19e1859b-a75a-4d37-af5c-e6e0bce0921b" containerID="b09e27df9716bf743171cd94707803ed8eed14537766965d8fbe7dbf8a993570" exitCode=137 Nov 03 02:14:35.117473 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:35.116054 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7011-7ece81" Nov 03 02:14:35.117473 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:35.116088 5437 scope.go:122] "RemoveContainer" containerID="b09e27df9716bf743171cd94707803ed8eed14537766965d8fbe7dbf8a993570" Nov 03 02:14:35.117473 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:35.116286 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7011-4c2838" podUID="d3ceb6c5-67db-4e65-8850-7daee812cf22" containerName="test-curl-7011-4c2838" containerID="cri-o://b70d9040756b856c50a0c7230872a5f3ebdd0041fe150e0fd5c3ead74fd6a450" gracePeriod=2 Nov 03 02:14:35.127415 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:35.127390 5437 scope.go:122] "RemoveContainer" containerID="b09e27df9716bf743171cd94707803ed8eed14537766965d8fbe7dbf8a993570" Nov 03 02:14:35.127688 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:14:35.127669 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b09e27df9716bf743171cd94707803ed8eed14537766965d8fbe7dbf8a993570\": container with ID starting with b09e27df9716bf743171cd94707803ed8eed14537766965d8fbe7dbf8a993570 not found: ID does not exist" containerID="b09e27df9716bf743171cd94707803ed8eed14537766965d8fbe7dbf8a993570" Nov 03 02:14:35.127740 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:35.127696 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b09e27df9716bf743171cd94707803ed8eed14537766965d8fbe7dbf8a993570"} err="failed to get container status \"b09e27df9716bf743171cd94707803ed8eed14537766965d8fbe7dbf8a993570\": rpc error: code = NotFound desc = could not find container \"b09e27df9716bf743171cd94707803ed8eed14537766965d8fbe7dbf8a993570\": container with ID starting with b09e27df9716bf743171cd94707803ed8eed14537766965d8fbe7dbf8a993570 not found: ID does not exist" Nov 03 02:14:35.146531 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:35.146507 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95rws\" (UniqueName: \"kubernetes.io/projected/0346b8eb-e18f-4689-a880-71954310ace0-kube-api-access-95rws\") pod \"test-curl-7011-700993\" (UID: \"0346b8eb-e18f-4689-a880-71954310ace0\") " pod="opendatahub/test-curl-7011-700993" Nov 03 02:14:35.169694 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:14:35.169659 5437 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7011-4c2838\" is forbidden: User \"system:node:ip-10-0-136-16.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-16.us-west-2.compute.internal' and this object" podUID="d3ceb6c5-67db-4e65-8850-7daee812cf22" pod="opendatahub/test-curl-7011-4c2838" Nov 03 02:14:35.209299 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:35.209266 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95rws\" (UniqueName: \"kubernetes.io/projected/0346b8eb-e18f-4689-a880-71954310ace0-kube-api-access-95rws\") pod \"test-curl-7011-700993\" (UID: \"0346b8eb-e18f-4689-a880-71954310ace0\") " pod="opendatahub/test-curl-7011-700993" Nov 03 02:14:35.222558 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:14:35.222526 5437 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7011-4c2838\" is forbidden: User \"system:node:ip-10-0-136-16.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-16.us-west-2.compute.internal' and this object" podUID="d3ceb6c5-67db-4e65-8850-7daee812cf22" pod="opendatahub/test-curl-7011-4c2838" Nov 03 02:14:35.245784 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:35.245757 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7011-700993" Nov 03 02:14:35.275343 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:14:35.275305 5437 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7011-7ece81\" is forbidden: User \"system:node:ip-10-0-136-16.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-16.us-west-2.compute.internal' and this object" podUID="19e1859b-a75a-4d37-af5c-e6e0bce0921b" pod="opendatahub/test-curl-7011-7ece81" Nov 03 02:14:35.617415 ip-10-0-136-16 kubenswrapper[5437]: W1103 02:14:35.617346 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0346b8eb_e18f_4689_a880_71954310ace0.slice/crio-cccde6e0a8281e644987fed68e7bc3493b1414ec372bd08cbf21d6373be1eef4 WatchSource:0}: Error finding container cccde6e0a8281e644987fed68e7bc3493b1414ec372bd08cbf21d6373be1eef4: Status 404 returned error can't find the container with id cccde6e0a8281e644987fed68e7bc3493b1414ec372bd08cbf21d6373be1eef4 Nov 03 02:14:35.621011 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:35.620987 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7011-700993"] Nov 03 02:14:36.089674 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:36.089638 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19e1859b-a75a-4d37-af5c-e6e0bce0921b" path="/var/lib/kubelet/pods/19e1859b-a75a-4d37-af5c-e6e0bce0921b/volumes" Nov 03 02:14:36.090275 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:36.090248 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2740d82-3082-474a-8726-4aaaecf57272" path="/var/lib/kubelet/pods/b2740d82-3082-474a-8726-4aaaecf57272/volumes" Nov 03 02:14:36.123284 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:36.123250 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7011-700993" event={"ID":"0346b8eb-e18f-4689-a880-71954310ace0","Type":"ContainerStarted","Data":"914bd034f934dd2ef4195171d86fa518c9e5ce079f92a08da98c53e3557b0159"} Nov 03 02:14:36.123733 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:36.123300 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7011-700993" event={"ID":"0346b8eb-e18f-4689-a880-71954310ace0","Type":"ContainerStarted","Data":"cccde6e0a8281e644987fed68e7bc3493b1414ec372bd08cbf21d6373be1eef4"} Nov 03 02:14:36.126327 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:36.126294 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-059b23/auth-test-model-059b23-kserve-797887d945-j9d4h" podUID="bf53e145-1d93-42fe-84eb-116775ebb4d4" containerName="main" containerID="cri-o://edac66301490ec37937eb8c10dc3ae7cf6c71bb959c63d8cc22049b1e9182e1f" gracePeriod=45 Nov 03 02:14:36.241600 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:36.241548 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7011-700993" podStartSLOduration=2.241532678 podStartE2EDuration="2.241532678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:14:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:14:36.239746238 +0000 UTC m=+3358.269228533" watchObservedRunningTime="2026-11-03 02:14:36.241532678 +0000 UTC m=+3358.271014973" Nov 03 02:14:36.398715 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:36.398686 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-059b23/auth-test-model-059b23-kserve-797887d945-j9d4h" Nov 03 02:14:36.462401 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:36.462353 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf53e145-1d93-42fe-84eb-116775ebb4d4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf53e145-1d93-42fe-84eb-116775ebb4d4-model-cache\") pod \"bf53e145-1d93-42fe-84eb-116775ebb4d4\" (UID: \"bf53e145-1d93-42fe-84eb-116775ebb4d4\") " Nov 03 02:14:36.462401 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:36.462406 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf53e145-1d93-42fe-84eb-116775ebb4d4-tls-certs\" (UniqueName: \"kubernetes.io/secret/bf53e145-1d93-42fe-84eb-116775ebb4d4-tls-certs\") pod \"bf53e145-1d93-42fe-84eb-116775ebb4d4\" (UID: \"bf53e145-1d93-42fe-84eb-116775ebb4d4\") " Nov 03 02:14:36.462655 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:36.462519 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf53e145-1d93-42fe-84eb-116775ebb4d4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf53e145-1d93-42fe-84eb-116775ebb4d4-dshm\") pod \"bf53e145-1d93-42fe-84eb-116775ebb4d4\" (UID: \"bf53e145-1d93-42fe-84eb-116775ebb4d4\") " Nov 03 02:14:36.462655 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:36.462619 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf53e145-1d93-42fe-84eb-116775ebb4d4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf53e145-1d93-42fe-84eb-116775ebb4d4-tmp-dir\") pod \"bf53e145-1d93-42fe-84eb-116775ebb4d4\" (UID: \"bf53e145-1d93-42fe-84eb-116775ebb4d4\") " Nov 03 02:14:36.462763 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:36.462664 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf53e145-1d93-42fe-84eb-116775ebb4d4-model-cache" pod "bf53e145-1d93-42fe-84eb-116775ebb4d4" (UID: "bf53e145-1d93-42fe-84eb-116775ebb4d4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:14:36.462763 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:36.462673 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf53e145-1d93-42fe-84eb-116775ebb4d4-home\" (UniqueName: \"kubernetes.io/empty-dir/bf53e145-1d93-42fe-84eb-116775ebb4d4-home\") pod \"bf53e145-1d93-42fe-84eb-116775ebb4d4\" (UID: \"bf53e145-1d93-42fe-84eb-116775ebb4d4\") " Nov 03 02:14:36.462763 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:36.462742 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf53e145-1d93-42fe-84eb-116775ebb4d4-kube-api-access-mtqll\" (UniqueName: \"kubernetes.io/projected/bf53e145-1d93-42fe-84eb-116775ebb4d4-kube-api-access-mtqll\") pod \"bf53e145-1d93-42fe-84eb-116775ebb4d4\" (UID: \"bf53e145-1d93-42fe-84eb-116775ebb4d4\") " Nov 03 02:14:36.462925 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:36.462823 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf53e145-1d93-42fe-84eb-116775ebb4d4-tmp-dir" pod "bf53e145-1d93-42fe-84eb-116775ebb4d4" (UID: "bf53e145-1d93-42fe-84eb-116775ebb4d4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:14:36.462925 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:36.462869 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf53e145-1d93-42fe-84eb-116775ebb4d4-home" pod "bf53e145-1d93-42fe-84eb-116775ebb4d4" (UID: "bf53e145-1d93-42fe-84eb-116775ebb4d4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:14:36.463317 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:36.463292 5437 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf53e145-1d93-42fe-84eb-116775ebb4d4-tmp-dir\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:14:36.463317 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:36.463317 5437 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf53e145-1d93-42fe-84eb-116775ebb4d4-home\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:14:36.463588 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:36.463326 5437 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf53e145-1d93-42fe-84eb-116775ebb4d4-model-cache\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:14:36.465026 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:36.464997 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf53e145-1d93-42fe-84eb-116775ebb4d4-tls-certs" pod "bf53e145-1d93-42fe-84eb-116775ebb4d4" (UID: "bf53e145-1d93-42fe-84eb-116775ebb4d4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:14:36.465134 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:36.465079 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf53e145-1d93-42fe-84eb-116775ebb4d4-dshm" pod "bf53e145-1d93-42fe-84eb-116775ebb4d4" (UID: "bf53e145-1d93-42fe-84eb-116775ebb4d4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:14:36.465134 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:36.465092 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf53e145-1d93-42fe-84eb-116775ebb4d4-kube-api-access-mtqll" pod "bf53e145-1d93-42fe-84eb-116775ebb4d4" (UID: "bf53e145-1d93-42fe-84eb-116775ebb4d4"). InnerVolumeSpecName "kube-api-access-mtqll". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:14:36.564813 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:36.564777 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtqll\" (UniqueName: \"kubernetes.io/projected/bf53e145-1d93-42fe-84eb-116775ebb4d4-kube-api-access-mtqll\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:14:36.564813 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:36.564810 5437 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf53e145-1d93-42fe-84eb-116775ebb4d4-tls-certs\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:14:36.564813 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:36.564820 5437 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf53e145-1d93-42fe-84eb-116775ebb4d4-dshm\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:14:36.787008 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:36.786927 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-8hvq2"] Nov 03 02:14:36.787172 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:36.787133 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-8hvq2" podUID="d287f009-3621-4858-9eed-5b806f637d6b" containerName="istio-proxy" containerID="cri-o://3c8b1c781f4d9c03c6536661c2217530b97662d1a785f39cc23bf2241bfb7914" gracePeriod=30 Nov 03 02:14:37.132455 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:37.132425 5437 generic.go:356] "Generic (PLEG): container finished" podID="bf53e145-1d93-42fe-84eb-116775ebb4d4" containerID="edac66301490ec37937eb8c10dc3ae7cf6c71bb959c63d8cc22049b1e9182e1f" exitCode=0 Nov 03 02:14:37.132880 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:37.132473 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-059b23/auth-test-model-059b23-kserve-797887d945-j9d4h" Nov 03 02:14:37.132880 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:37.132506 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-059b23/auth-test-model-059b23-kserve-797887d945-j9d4h" event={"ID":"bf53e145-1d93-42fe-84eb-116775ebb4d4","Type":"ContainerDied","Data":"edac66301490ec37937eb8c10dc3ae7cf6c71bb959c63d8cc22049b1e9182e1f"} Nov 03 02:14:37.132880 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:37.132564 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-059b23/auth-test-model-059b23-kserve-797887d945-j9d4h" event={"ID":"bf53e145-1d93-42fe-84eb-116775ebb4d4","Type":"ContainerDied","Data":"8f47d8e56101c0cb42fee381e62aa6436821037507366a169256e5821a39ebfe"} Nov 03 02:14:37.132880 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:37.132592 5437 scope.go:122] "RemoveContainer" containerID="edac66301490ec37937eb8c10dc3ae7cf6c71bb959c63d8cc22049b1e9182e1f" Nov 03 02:14:37.203584 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:37.203555 5437 scope.go:122] "RemoveContainer" containerID="edac66301490ec37937eb8c10dc3ae7cf6c71bb959c63d8cc22049b1e9182e1f" Nov 03 02:14:37.203954 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:14:37.203926 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edac66301490ec37937eb8c10dc3ae7cf6c71bb959c63d8cc22049b1e9182e1f\": container with ID starting with edac66301490ec37937eb8c10dc3ae7cf6c71bb959c63d8cc22049b1e9182e1f not found: ID does not exist" containerID="edac66301490ec37937eb8c10dc3ae7cf6c71bb959c63d8cc22049b1e9182e1f" Nov 03 02:14:37.204026 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:37.203965 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edac66301490ec37937eb8c10dc3ae7cf6c71bb959c63d8cc22049b1e9182e1f"} err="failed to get container status \"edac66301490ec37937eb8c10dc3ae7cf6c71bb959c63d8cc22049b1e9182e1f\": rpc error: code = NotFound desc = could not find container \"edac66301490ec37937eb8c10dc3ae7cf6c71bb959c63d8cc22049b1e9182e1f\": container with ID starting with edac66301490ec37937eb8c10dc3ae7cf6c71bb959c63d8cc22049b1e9182e1f not found: ID does not exist" Nov 03 02:14:37.233187 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:37.233150 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:14:37.312776 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:37.312742 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-059b23/auth-test-model-059b23-kserve-797887d945-j9d4h"] Nov 03 02:14:37.317316 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:37.317291 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-059b23/auth-test-model-059b23-kserve-797887d945-j9d4h"] Nov 03 02:14:37.343803 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:37.343776 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7011-4c2838" Nov 03 02:14:37.477994 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:37.477959 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3ceb6c5-67db-4e65-8850-7daee812cf22-kube-api-access-jwl2j\" (UniqueName: \"kubernetes.io/projected/d3ceb6c5-67db-4e65-8850-7daee812cf22-kube-api-access-jwl2j\") pod \"d3ceb6c5-67db-4e65-8850-7daee812cf22\" (UID: \"d3ceb6c5-67db-4e65-8850-7daee812cf22\") " Nov 03 02:14:37.480040 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:37.480012 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3ceb6c5-67db-4e65-8850-7daee812cf22-kube-api-access-jwl2j" pod "d3ceb6c5-67db-4e65-8850-7daee812cf22" (UID: "d3ceb6c5-67db-4e65-8850-7daee812cf22"). InnerVolumeSpecName "kube-api-access-jwl2j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:14:37.579896 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:37.579861 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwl2j\" (UniqueName: \"kubernetes.io/projected/d3ceb6c5-67db-4e65-8850-7daee812cf22-kube-api-access-jwl2j\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:14:37.684688 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:37.684592 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7011-700993"] Nov 03 02:14:37.690433 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:37.690406 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7011-700993"] Nov 03 02:14:37.829213 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:37.829180 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7011-9747e1"] Nov 03 02:14:37.829844 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:37.829823 5437 cpu_manager.go:414] "containerMap: removing container" podUID="bf53e145-1d93-42fe-84eb-116775ebb4d4" containerName="main" Nov 03 02:14:37.829935 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:37.829846 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf53e145-1d93-42fe-84eb-116775ebb4d4" containerName="main" Nov 03 02:14:37.829935 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:37.829878 5437 cpu_manager.go:414] "containerMap: removing container" podUID="0346b8eb-e18f-4689-a880-71954310ace0" containerName="test-curl-7011-700993" Nov 03 02:14:37.829935 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:37.829886 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0346b8eb-e18f-4689-a880-71954310ace0" containerName="test-curl-7011-700993" Nov 03 02:14:37.830103 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:37.830085 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf53e145-1d93-42fe-84eb-116775ebb4d4" containerName="main" Nov 03 02:14:37.830154 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:37.830106 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="0346b8eb-e18f-4689-a880-71954310ace0" containerName="test-curl-7011-700993" Nov 03 02:14:37.833774 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:37.833752 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7011-9747e1" Nov 03 02:14:37.883068 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:37.883038 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7011-9747e1"] Nov 03 02:14:37.984101 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:37.984015 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptwfh\" (UniqueName: \"kubernetes.io/projected/fca1825b-eccc-4163-b0f0-d55c02c92fb7-kube-api-access-ptwfh\") pod \"test-curl-7011-9747e1\" (UID: \"fca1825b-eccc-4163-b0f0-d55c02c92fb7\") " pod="opendatahub/test-curl-7011-9747e1" Nov 03 02:14:38.084946 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:38.084912 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptwfh\" (UniqueName: \"kubernetes.io/projected/fca1825b-eccc-4163-b0f0-d55c02c92fb7-kube-api-access-ptwfh\") pod \"test-curl-7011-9747e1\" (UID: \"fca1825b-eccc-4163-b0f0-d55c02c92fb7\") " pod="opendatahub/test-curl-7011-9747e1" Nov 03 02:14:38.090877 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:38.090847 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf53e145-1d93-42fe-84eb-116775ebb4d4" path="/var/lib/kubelet/pods/bf53e145-1d93-42fe-84eb-116775ebb4d4/volumes" Nov 03 02:14:38.091218 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:38.091204 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3ceb6c5-67db-4e65-8850-7daee812cf22" path="/var/lib/kubelet/pods/d3ceb6c5-67db-4e65-8850-7daee812cf22/volumes" Nov 03 02:14:38.140561 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:38.140522 5437 generic.go:356] "Generic (PLEG): container finished" podID="d3ceb6c5-67db-4e65-8850-7daee812cf22" containerID="b70d9040756b856c50a0c7230872a5f3ebdd0041fe150e0fd5c3ead74fd6a450" exitCode=137 Nov 03 02:14:38.140969 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:38.140582 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7011-4c2838" Nov 03 02:14:38.140969 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:38.140601 5437 scope.go:122] "RemoveContainer" containerID="b70d9040756b856c50a0c7230872a5f3ebdd0041fe150e0fd5c3ead74fd6a450" Nov 03 02:14:38.141859 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:38.141789 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7011-700993" podUID="0346b8eb-e18f-4689-a880-71954310ace0" containerName="test-curl-7011-700993" containerID="cri-o://914bd034f934dd2ef4195171d86fa518c9e5ce079f92a08da98c53e3557b0159" gracePeriod=2 Nov 03 02:14:38.147620 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:38.147598 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptwfh\" (UniqueName: \"kubernetes.io/projected/fca1825b-eccc-4163-b0f0-d55c02c92fb7-kube-api-access-ptwfh\") pod \"test-curl-7011-9747e1\" (UID: \"fca1825b-eccc-4163-b0f0-d55c02c92fb7\") " pod="opendatahub/test-curl-7011-9747e1" Nov 03 02:14:38.151997 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:38.151976 5437 scope.go:122] "RemoveContainer" containerID="b70d9040756b856c50a0c7230872a5f3ebdd0041fe150e0fd5c3ead74fd6a450" Nov 03 02:14:38.152624 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:14:38.152596 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b70d9040756b856c50a0c7230872a5f3ebdd0041fe150e0fd5c3ead74fd6a450\": container with ID starting with b70d9040756b856c50a0c7230872a5f3ebdd0041fe150e0fd5c3ead74fd6a450 not found: ID does not exist" containerID="b70d9040756b856c50a0c7230872a5f3ebdd0041fe150e0fd5c3ead74fd6a450" Nov 03 02:14:38.152786 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:38.152751 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b70d9040756b856c50a0c7230872a5f3ebdd0041fe150e0fd5c3ead74fd6a450"} err="failed to get container status \"b70d9040756b856c50a0c7230872a5f3ebdd0041fe150e0fd5c3ead74fd6a450\": rpc error: code = NotFound desc = could not find container \"b70d9040756b856c50a0c7230872a5f3ebdd0041fe150e0fd5c3ead74fd6a450\": container with ID starting with b70d9040756b856c50a0c7230872a5f3ebdd0041fe150e0fd5c3ead74fd6a450 not found: ID does not exist" Nov 03 02:14:38.164461 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:38.164424 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:14:38.446842 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:38.446807 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7011-9747e1" Nov 03 02:14:38.820046 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:38.820017 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7011-9747e1"] Nov 03 02:14:39.149061 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:39.149016 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7011-9747e1" event={"ID":"fca1825b-eccc-4163-b0f0-d55c02c92fb7","Type":"ContainerStarted","Data":"3a6c9ac5d0fbd12aa1950682e863b3ee542b635446d232d2eda9b2dfaca9fca2"} Nov 03 02:14:39.149061 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:39.149067 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7011-9747e1" event={"ID":"fca1825b-eccc-4163-b0f0-d55c02c92fb7","Type":"ContainerStarted","Data":"93ded7df573e27ca9fd8d085a1fd51e30c8dd11c41c43ad32fe61f80972bb3d4"} Nov 03 02:14:39.268438 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:39.268379 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7011-9747e1" podStartSLOduration=2.268354862 podStartE2EDuration="2.268354862s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:14:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:14:39.265696036 +0000 UTC m=+3361.295178330" watchObservedRunningTime="2026-11-03 02:14:39.268354862 +0000 UTC m=+3361.297837157" Nov 03 02:14:40.150917 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:14:40.150881 5437 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0346b8eb_e18f_4689_a880_71954310ace0.slice/crio-cccde6e0a8281e644987fed68e7bc3493b1414ec372bd08cbf21d6373be1eef4\": RecentStats: unable to find data in memory cache]" Nov 03 02:14:40.377734 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:40.377706 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7011-700993" Nov 03 02:14:40.515323 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:40.515242 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0346b8eb-e18f-4689-a880-71954310ace0-kube-api-access-95rws\" (UniqueName: \"kubernetes.io/projected/0346b8eb-e18f-4689-a880-71954310ace0-kube-api-access-95rws\") pod \"0346b8eb-e18f-4689-a880-71954310ace0\" (UID: \"0346b8eb-e18f-4689-a880-71954310ace0\") " Nov 03 02:14:40.517317 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:40.517282 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0346b8eb-e18f-4689-a880-71954310ace0-kube-api-access-95rws" pod "0346b8eb-e18f-4689-a880-71954310ace0" (UID: "0346b8eb-e18f-4689-a880-71954310ace0"). InnerVolumeSpecName "kube-api-access-95rws". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:14:40.616803 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:40.616768 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95rws\" (UniqueName: \"kubernetes.io/projected/0346b8eb-e18f-4689-a880-71954310ace0-kube-api-access-95rws\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:14:40.805970 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:40.805936 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7011-9747e1"] Nov 03 02:14:40.809923 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:40.809885 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7011-9747e1"] Nov 03 02:14:41.164206 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:41.164104 5437 generic.go:356] "Generic (PLEG): container finished" podID="0346b8eb-e18f-4689-a880-71954310ace0" containerID="914bd034f934dd2ef4195171d86fa518c9e5ce079f92a08da98c53e3557b0159" exitCode=137 Nov 03 02:14:41.164206 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:41.164154 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7011-700993" Nov 03 02:14:41.164206 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:41.164203 5437 scope.go:122] "RemoveContainer" containerID="914bd034f934dd2ef4195171d86fa518c9e5ce079f92a08da98c53e3557b0159" Nov 03 02:14:41.164814 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:41.164464 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7011-9747e1" podUID="fca1825b-eccc-4163-b0f0-d55c02c92fb7" containerName="test-curl-7011-9747e1" containerID="cri-o://3a6c9ac5d0fbd12aa1950682e863b3ee542b635446d232d2eda9b2dfaca9fca2" gracePeriod=2 Nov 03 02:14:41.173485 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:41.173467 5437 scope.go:122] "RemoveContainer" containerID="914bd034f934dd2ef4195171d86fa518c9e5ce079f92a08da98c53e3557b0159" Nov 03 02:14:41.173753 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:14:41.173735 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"914bd034f934dd2ef4195171d86fa518c9e5ce079f92a08da98c53e3557b0159\": container with ID starting with 914bd034f934dd2ef4195171d86fa518c9e5ce079f92a08da98c53e3557b0159 not found: ID does not exist" containerID="914bd034f934dd2ef4195171d86fa518c9e5ce079f92a08da98c53e3557b0159" Nov 03 02:14:41.173829 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:41.173763 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"914bd034f934dd2ef4195171d86fa518c9e5ce079f92a08da98c53e3557b0159"} err="failed to get container status \"914bd034f934dd2ef4195171d86fa518c9e5ce079f92a08da98c53e3557b0159\": rpc error: code = NotFound desc = could not find container \"914bd034f934dd2ef4195171d86fa518c9e5ce079f92a08da98c53e3557b0159\": container with ID starting with 914bd034f934dd2ef4195171d86fa518c9e5ce079f92a08da98c53e3557b0159 not found: ID does not exist" Nov 03 02:14:41.217702 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:14:41.217668 5437 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7011-9747e1\" is forbidden: User \"system:node:ip-10-0-136-16.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-16.us-west-2.compute.internal' and this object" podUID="fca1825b-eccc-4163-b0f0-d55c02c92fb7" pod="opendatahub/test-curl-7011-9747e1" Nov 03 02:14:41.270620 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:14:41.270584 5437 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7011-9747e1\" is forbidden: User \"system:node:ip-10-0-136-16.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-16.us-west-2.compute.internal' and this object" podUID="fca1825b-eccc-4163-b0f0-d55c02c92fb7" pod="opendatahub/test-curl-7011-9747e1" Nov 03 02:14:41.323464 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:14:41.323422 5437 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7011-700993\" is forbidden: User \"system:node:ip-10-0-136-16.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-16.us-west-2.compute.internal' and this object" podUID="0346b8eb-e18f-4689-a880-71954310ace0" pod="opendatahub/test-curl-7011-700993" Nov 03 02:14:42.064178 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:42.064112 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-8hvq2" Nov 03 02:14:42.088844 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:42.088813 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0346b8eb-e18f-4689-a880-71954310ace0" path="/var/lib/kubelet/pods/0346b8eb-e18f-4689-a880-71954310ace0/volumes" Nov 03 02:14:42.117136 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:14:42.117100 5437 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7011-700993\" is forbidden: User \"system:node:ip-10-0-136-16.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-16.us-west-2.compute.internal' and this object" podUID="0346b8eb-e18f-4689-a880-71954310ace0" pod="opendatahub/test-curl-7011-700993" Nov 03 02:14:42.173098 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:42.173058 5437 generic.go:356] "Generic (PLEG): container finished" podID="d287f009-3621-4858-9eed-5b806f637d6b" containerID="3c8b1c781f4d9c03c6536661c2217530b97662d1a785f39cc23bf2241bfb7914" exitCode=0 Nov 03 02:14:42.173547 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:42.173116 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-8hvq2" Nov 03 02:14:42.173547 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:42.173145 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-8hvq2" event={"ID":"d287f009-3621-4858-9eed-5b806f637d6b","Type":"ContainerDied","Data":"3c8b1c781f4d9c03c6536661c2217530b97662d1a785f39cc23bf2241bfb7914"} Nov 03 02:14:42.173547 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:42.173203 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-8hvq2" event={"ID":"d287f009-3621-4858-9eed-5b806f637d6b","Type":"ContainerDied","Data":"39f73ecb17f12492717f7e1253a722df3e9b2ca109263a22eec3eac254238766"} Nov 03 02:14:42.173547 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:42.173226 5437 scope.go:122] "RemoveContainer" containerID="3c8b1c781f4d9c03c6536661c2217530b97662d1a785f39cc23bf2241bfb7914" Nov 03 02:14:42.182775 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:42.182757 5437 scope.go:122] "RemoveContainer" containerID="3c8b1c781f4d9c03c6536661c2217530b97662d1a785f39cc23bf2241bfb7914" Nov 03 02:14:42.183042 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:14:42.183020 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c8b1c781f4d9c03c6536661c2217530b97662d1a785f39cc23bf2241bfb7914\": container with ID starting with 3c8b1c781f4d9c03c6536661c2217530b97662d1a785f39cc23bf2241bfb7914 not found: ID does not exist" containerID="3c8b1c781f4d9c03c6536661c2217530b97662d1a785f39cc23bf2241bfb7914" Nov 03 02:14:42.183087 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:42.183052 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c8b1c781f4d9c03c6536661c2217530b97662d1a785f39cc23bf2241bfb7914"} err="failed to get container status \"3c8b1c781f4d9c03c6536661c2217530b97662d1a785f39cc23bf2241bfb7914\": rpc error: code = NotFound desc = could not find container \"3c8b1c781f4d9c03c6536661c2217530b97662d1a785f39cc23bf2241bfb7914\": container with ID starting with 3c8b1c781f4d9c03c6536661c2217530b97662d1a785f39cc23bf2241bfb7914 not found: ID does not exist" Nov 03 02:14:42.245183 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:42.245146 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d287f009-3621-4858-9eed-5b806f637d6b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d287f009-3621-4858-9eed-5b806f637d6b-workload-certs\") pod \"d287f009-3621-4858-9eed-5b806f637d6b\" (UID: \"d287f009-3621-4858-9eed-5b806f637d6b\") " Nov 03 02:14:42.245348 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:42.245276 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d287f009-3621-4858-9eed-5b806f637d6b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d287f009-3621-4858-9eed-5b806f637d6b-credential-socket\") pod \"d287f009-3621-4858-9eed-5b806f637d6b\" (UID: \"d287f009-3621-4858-9eed-5b806f637d6b\") " Nov 03 02:14:42.245348 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:42.245331 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d287f009-3621-4858-9eed-5b806f637d6b-kube-api-access-4qddk\" (UniqueName: \"kubernetes.io/projected/d287f009-3621-4858-9eed-5b806f637d6b-kube-api-access-4qddk\") pod \"d287f009-3621-4858-9eed-5b806f637d6b\" (UID: \"d287f009-3621-4858-9eed-5b806f637d6b\") " Nov 03 02:14:42.245466 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:42.245381 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d287f009-3621-4858-9eed-5b806f637d6b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d287f009-3621-4858-9eed-5b806f637d6b-workload-socket\") pod \"d287f009-3621-4858-9eed-5b806f637d6b\" (UID: \"d287f009-3621-4858-9eed-5b806f637d6b\") " Nov 03 02:14:42.245466 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:42.245441 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d287f009-3621-4858-9eed-5b806f637d6b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d287f009-3621-4858-9eed-5b806f637d6b-istiod-ca-cert\") pod \"d287f009-3621-4858-9eed-5b806f637d6b\" (UID: \"d287f009-3621-4858-9eed-5b806f637d6b\") " Nov 03 02:14:42.245575 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:42.245518 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d287f009-3621-4858-9eed-5b806f637d6b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d287f009-3621-4858-9eed-5b806f637d6b-istio-envoy\") pod \"d287f009-3621-4858-9eed-5b806f637d6b\" (UID: \"d287f009-3621-4858-9eed-5b806f637d6b\") " Nov 03 02:14:42.245575 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:42.245520 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d287f009-3621-4858-9eed-5b806f637d6b-workload-certs" pod "d287f009-3621-4858-9eed-5b806f637d6b" (UID: "d287f009-3621-4858-9eed-5b806f637d6b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:14:42.245685 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:42.245594 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d287f009-3621-4858-9eed-5b806f637d6b-istio-token\" (UniqueName: \"kubernetes.io/projected/d287f009-3621-4858-9eed-5b806f637d6b-istio-token\") pod \"d287f009-3621-4858-9eed-5b806f637d6b\" (UID: \"d287f009-3621-4858-9eed-5b806f637d6b\") " Nov 03 02:14:42.245685 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:42.245639 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d287f009-3621-4858-9eed-5b806f637d6b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d287f009-3621-4858-9eed-5b806f637d6b-istio-data\") pod \"d287f009-3621-4858-9eed-5b806f637d6b\" (UID: \"d287f009-3621-4858-9eed-5b806f637d6b\") " Nov 03 02:14:42.245685 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:42.245669 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d287f009-3621-4858-9eed-5b806f637d6b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d287f009-3621-4858-9eed-5b806f637d6b-istio-podinfo\") pod \"d287f009-3621-4858-9eed-5b806f637d6b\" (UID: \"d287f009-3621-4858-9eed-5b806f637d6b\") " Nov 03 02:14:42.246136 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:42.245917 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d287f009-3621-4858-9eed-5b806f637d6b-istiod-ca-cert" pod "d287f009-3621-4858-9eed-5b806f637d6b" (UID: "d287f009-3621-4858-9eed-5b806f637d6b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:14:42.246235 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:42.246137 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d287f009-3621-4858-9eed-5b806f637d6b-workload-socket" pod "d287f009-3621-4858-9eed-5b806f637d6b" (UID: "d287f009-3621-4858-9eed-5b806f637d6b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:14:42.246235 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:42.246169 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d287f009-3621-4858-9eed-5b806f637d6b-istio-data" pod "d287f009-3621-4858-9eed-5b806f637d6b" (UID: "d287f009-3621-4858-9eed-5b806f637d6b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:14:42.246401 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:42.246335 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d287f009-3621-4858-9eed-5b806f637d6b-credential-socket" pod "d287f009-3621-4858-9eed-5b806f637d6b" (UID: "d287f009-3621-4858-9eed-5b806f637d6b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:14:42.246572 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:42.246485 5437 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d287f009-3621-4858-9eed-5b806f637d6b-workload-socket\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:14:42.246572 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:42.246523 5437 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d287f009-3621-4858-9eed-5b806f637d6b-istiod-ca-cert\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:14:42.246572 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:42.246537 5437 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d287f009-3621-4858-9eed-5b806f637d6b-istio-data\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:14:42.246572 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:42.246552 5437 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d287f009-3621-4858-9eed-5b806f637d6b-workload-certs\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:14:42.248554 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:42.248512 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d287f009-3621-4858-9eed-5b806f637d6b-istio-envoy" pod "d287f009-3621-4858-9eed-5b806f637d6b" (UID: "d287f009-3621-4858-9eed-5b806f637d6b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:14:42.248801 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:42.248777 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d287f009-3621-4858-9eed-5b806f637d6b-istio-token" pod "d287f009-3621-4858-9eed-5b806f637d6b" (UID: "d287f009-3621-4858-9eed-5b806f637d6b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:14:42.248891 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:42.248799 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d287f009-3621-4858-9eed-5b806f637d6b-kube-api-access-4qddk" pod "d287f009-3621-4858-9eed-5b806f637d6b" (UID: "d287f009-3621-4858-9eed-5b806f637d6b"). InnerVolumeSpecName "kube-api-access-4qddk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:14:42.249159 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:42.249135 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d287f009-3621-4858-9eed-5b806f637d6b-istio-podinfo" pod "d287f009-3621-4858-9eed-5b806f637d6b" (UID: "d287f009-3621-4858-9eed-5b806f637d6b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:14:42.345055 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:14:42.344956 5437 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7011-9747e1\" is forbidden: User \"system:node:ip-10-0-136-16.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-16.us-west-2.compute.internal' and this object" podUID="fca1825b-eccc-4163-b0f0-d55c02c92fb7" pod="opendatahub/test-curl-7011-9747e1" Nov 03 02:14:42.347715 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:42.347680 5437 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d287f009-3621-4858-9eed-5b806f637d6b-credential-socket\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:14:42.347715 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:42.347714 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qddk\" (UniqueName: \"kubernetes.io/projected/d287f009-3621-4858-9eed-5b806f637d6b-kube-api-access-4qddk\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:14:42.347923 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:42.347731 5437 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d287f009-3621-4858-9eed-5b806f637d6b-istio-envoy\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:14:42.347923 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:42.347744 5437 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d287f009-3621-4858-9eed-5b806f637d6b-istio-token\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:14:42.347923 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:42.347756 5437 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d287f009-3621-4858-9eed-5b806f637d6b-istio-podinfo\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:14:42.385169 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:42.385102 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:14:42.655464 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:42.655379 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-8hvq2"] Nov 03 02:14:42.676247 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:42.676216 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-8hvq2"] Nov 03 02:14:42.720910 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:42.720864 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:14:43.041681 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:43.041558 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:14:43.401576 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:43.401547 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7011-9747e1" Nov 03 02:14:43.515166 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:43.515120 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:14:43.562337 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:43.562239 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fca1825b-eccc-4163-b0f0-d55c02c92fb7-kube-api-access-ptwfh\" (UniqueName: \"kubernetes.io/projected/fca1825b-eccc-4163-b0f0-d55c02c92fb7-kube-api-access-ptwfh\") pod \"fca1825b-eccc-4163-b0f0-d55c02c92fb7\" (UID: \"fca1825b-eccc-4163-b0f0-d55c02c92fb7\") " Nov 03 02:14:43.564405 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:43.564376 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fca1825b-eccc-4163-b0f0-d55c02c92fb7-kube-api-access-ptwfh" pod "fca1825b-eccc-4163-b0f0-d55c02c92fb7" (UID: "fca1825b-eccc-4163-b0f0-d55c02c92fb7"). InnerVolumeSpecName "kube-api-access-ptwfh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:14:43.663568 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:43.663529 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptwfh\" (UniqueName: \"kubernetes.io/projected/fca1825b-eccc-4163-b0f0-d55c02c92fb7-kube-api-access-ptwfh\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:14:43.847432 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:43.847315 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:14:44.035738 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:14:44.035701 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 02:14:44.089006 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:44.088959 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d287f009-3621-4858-9eed-5b806f637d6b" path="/var/lib/kubelet/pods/d287f009-3621-4858-9eed-5b806f637d6b/volumes" Nov 03 02:14:44.089460 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:44.089440 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fca1825b-eccc-4163-b0f0-d55c02c92fb7" path="/var/lib/kubelet/pods/fca1825b-eccc-4163-b0f0-d55c02c92fb7/volumes" Nov 03 02:14:44.187624 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:44.187523 5437 generic.go:356] "Generic (PLEG): container finished" podID="fca1825b-eccc-4163-b0f0-d55c02c92fb7" containerID="3a6c9ac5d0fbd12aa1950682e863b3ee542b635446d232d2eda9b2dfaca9fca2" exitCode=137 Nov 03 02:14:44.187624 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:44.187570 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7011-9747e1" Nov 03 02:14:44.187624 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:44.187623 5437 scope.go:122] "RemoveContainer" containerID="3a6c9ac5d0fbd12aa1950682e863b3ee542b635446d232d2eda9b2dfaca9fca2" Nov 03 02:14:44.196959 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:44.196812 5437 scope.go:122] "RemoveContainer" containerID="3a6c9ac5d0fbd12aa1950682e863b3ee542b635446d232d2eda9b2dfaca9fca2" Nov 03 02:14:44.197159 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:14:44.197130 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a6c9ac5d0fbd12aa1950682e863b3ee542b635446d232d2eda9b2dfaca9fca2\": container with ID starting with 3a6c9ac5d0fbd12aa1950682e863b3ee542b635446d232d2eda9b2dfaca9fca2 not found: ID does not exist" containerID="3a6c9ac5d0fbd12aa1950682e863b3ee542b635446d232d2eda9b2dfaca9fca2" Nov 03 02:14:44.197225 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:44.197170 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a6c9ac5d0fbd12aa1950682e863b3ee542b635446d232d2eda9b2dfaca9fca2"} err="failed to get container status \"3a6c9ac5d0fbd12aa1950682e863b3ee542b635446d232d2eda9b2dfaca9fca2\": rpc error: code = NotFound desc = could not find container \"3a6c9ac5d0fbd12aa1950682e863b3ee542b635446d232d2eda9b2dfaca9fca2\": container with ID starting with 3a6c9ac5d0fbd12aa1950682e863b3ee542b635446d232d2eda9b2dfaca9fca2 not found: ID does not exist" Nov 03 02:14:47.631536 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:47.631487 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:14:47.893846 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:47.893746 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:14:48.098642 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:48.098592 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:14:50.799588 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:50.799541 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:14:51.054327 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:51.054233 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:14:51.132028 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:51.131997 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-c70365-58bfdd8b7f-qw6f2"] Nov 03 02:14:51.132799 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:51.132772 5437 cpu_manager.go:414] "containerMap: removing container" podUID="d287f009-3621-4858-9eed-5b806f637d6b" containerName="istio-proxy" Nov 03 02:14:51.132799 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:51.132797 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d287f009-3621-4858-9eed-5b806f637d6b" containerName="istio-proxy" Nov 03 02:14:51.133021 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:51.132823 5437 cpu_manager.go:414] "containerMap: removing container" podUID="fca1825b-eccc-4163-b0f0-d55c02c92fb7" containerName="test-curl-7011-9747e1" Nov 03 02:14:51.133021 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:51.132828 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fca1825b-eccc-4163-b0f0-d55c02c92fb7" containerName="test-curl-7011-9747e1" Nov 03 02:14:51.133021 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:51.132954 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="fca1825b-eccc-4163-b0f0-d55c02c92fb7" containerName="test-curl-7011-9747e1" Nov 03 02:14:51.133021 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:51.132970 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="d287f009-3621-4858-9eed-5b806f637d6b" containerName="istio-proxy" Nov 03 02:14:51.139696 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:51.139672 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c70365-58bfdd8b7f-qw6f2" Nov 03 02:14:51.147233 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:51.146867 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5w52\" (UniqueName: \"kubernetes.io/projected/a5548a3f-3687-4453-ab8e-34f949a63e5a-kube-api-access-s5w52\") pod \"payload-pre-processing-e2e-extmod-a-c70365-58bfdd8b7f-qw6f2\" (UID: \"a5548a3f-3687-4453-ab8e-34f949a63e5a\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c70365-58bfdd8b7f-qw6f2" Nov 03 02:14:51.147233 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:51.147121 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5548a3f-3687-4453-ab8e-34f949a63e5a-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-c70365-58bfdd8b7f-qw6f2\" (UID: \"a5548a3f-3687-4453-ab8e-34f949a63e5a\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c70365-58bfdd8b7f-qw6f2" Nov 03 02:14:51.190421 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:51.190387 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-c70365-58bfdd8b7f-qw6f2"] Nov 03 02:14:51.239375 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:51.239329 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-c70365\"" Nov 03 02:14:51.239375 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:51.239328 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-c70365-dockercfg-vbpvs\"" Nov 03 02:14:51.255374 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:51.255263 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5548a3f-3687-4453-ab8e-34f949a63e5a-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-c70365-58bfdd8b7f-qw6f2\" (UID: \"a5548a3f-3687-4453-ab8e-34f949a63e5a\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c70365-58bfdd8b7f-qw6f2" Nov 03 02:14:51.255669 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:51.255482 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5w52\" (UniqueName: \"kubernetes.io/projected/a5548a3f-3687-4453-ab8e-34f949a63e5a-kube-api-access-s5w52\") pod \"payload-pre-processing-e2e-extmod-a-c70365-58bfdd8b7f-qw6f2\" (UID: \"a5548a3f-3687-4453-ab8e-34f949a63e5a\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c70365-58bfdd8b7f-qw6f2" Nov 03 02:14:51.256107 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:51.256077 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5548a3f-3687-4453-ab8e-34f949a63e5a-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-c70365-58bfdd8b7f-qw6f2\" (UID: \"a5548a3f-3687-4453-ab8e-34f949a63e5a\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c70365-58bfdd8b7f-qw6f2" Nov 03 02:14:51.320427 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:51.320344 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5w52\" (UniqueName: \"kubernetes.io/projected/a5548a3f-3687-4453-ab8e-34f949a63e5a-kube-api-access-s5w52\") pod \"payload-pre-processing-e2e-extmod-a-c70365-58bfdd8b7f-qw6f2\" (UID: \"a5548a3f-3687-4453-ab8e-34f949a63e5a\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c70365-58bfdd8b7f-qw6f2" Nov 03 02:14:51.454187 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:51.454148 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c70365-58bfdd8b7f-qw6f2" Nov 03 02:14:51.827911 ip-10-0-136-16 kubenswrapper[5437]: W1103 02:14:51.827877 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5548a3f_3687_4453_ab8e_34f949a63e5a.slice/crio-0b4af3fda6f78bfeef056329f0fc681867c7068cc29ea54b5e70e36e89e81803 WatchSource:0}: Error finding container 0b4af3fda6f78bfeef056329f0fc681867c7068cc29ea54b5e70e36e89e81803: Status 404 returned error can't find the container with id 0b4af3fda6f78bfeef056329f0fc681867c7068cc29ea54b5e70e36e89e81803 Nov 03 02:14:51.830608 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:51.830581 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-c70365-58bfdd8b7f-qw6f2"] Nov 03 02:14:52.240223 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:52.240123 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c70365-58bfdd8b7f-qw6f2" event={"ID":"a5548a3f-3687-4453-ab8e-34f949a63e5a","Type":"ContainerStarted","Data":"807bad0567ade55ce480f0bfd61aed104c6679f8d46c55f85fcfa3faa380b073"} Nov 03 02:14:52.240223 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:52.240184 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c70365-58bfdd8b7f-qw6f2" event={"ID":"a5548a3f-3687-4453-ab8e-34f949a63e5a","Type":"ContainerStarted","Data":"0b4af3fda6f78bfeef056329f0fc681867c7068cc29ea54b5e70e36e89e81803"} Nov 03 02:14:52.240223 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:52.240206 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c70365-58bfdd8b7f-qw6f2" Nov 03 02:14:52.359646 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:52.359596 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c70365-58bfdd8b7f-qw6f2" podStartSLOduration=1.35958187 podStartE2EDuration="1.35958187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:14:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:14:52.35958244 +0000 UTC m=+3374.389064735" watchObservedRunningTime="2026-11-03 02:14:52.35958187 +0000 UTC m=+3374.389064161" Nov 03 02:14:53.137058 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:53.137011 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:14:54.958538 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:54.958495 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:14:57.583093 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:57.583054 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:14:58.189167 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:58.189124 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-kzj4s"] Nov 03 02:14:58.189436 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:58.189410 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-kzj4s" podUID="ced2a84a-85cd-4435-bbca-33ee4ae19496" containerName="istio-proxy" containerID="cri-o://ba41559d0eeaf4d3a5557e8b0700d53f0c491d8b40c1fafeba2241cb14129d1d" gracePeriod=30 Nov 03 02:14:59.752618 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:14:59.752567 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:15:00.188682 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:00.188638 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e1793d-29894535-ndjl5"] Nov 03 02:15:00.195914 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:00.195885 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e1793d-29894535-ndjl5" Nov 03 02:15:00.212442 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:00.212406 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-b16721-29894535-jnxvl"] Nov 03 02:15:00.216819 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:00.216789 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-98a394-29894535-x2v6t"] Nov 03 02:15:00.216978 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:00.216941 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-b16721-29894535-jnxvl" Nov 03 02:15:00.221080 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:00.221054 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-98a394-29894535-x2v6t" Nov 03 02:15:00.264089 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:00.264054 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/437aab4b-3661-46d1-a41f-8da45f894600-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-e1793d-29894535-ndjl5\" (UID: \"437aab4b-3661-46d1-a41f-8da45f894600\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e1793d-29894535-ndjl5" Nov 03 02:15:00.264273 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:00.264155 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d82m2\" (UniqueName: \"kubernetes.io/projected/437aab4b-3661-46d1-a41f-8da45f894600-kube-api-access-d82m2\") pod \"maas-api-key-cleanup-e2e-worker-w1-e1793d-29894535-ndjl5\" (UID: \"437aab4b-3661-46d1-a41f-8da45f894600\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e1793d-29894535-ndjl5" Nov 03 02:15:00.264273 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:00.264188 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ad15281-5d31-4d89-bd48-185887526864-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-explicit-b16721-29894535-jnxvl\" (UID: \"6ad15281-5d31-4d89-bd48-185887526864\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-b16721-29894535-jnxvl" Nov 03 02:15:00.264273 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:00.264224 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7grrg\" (UniqueName: \"kubernetes.io/projected/6ad15281-5d31-4d89-bd48-185887526864-kube-api-access-7grrg\") pod \"maas-api-key-cleanup-e2e-explicit-b16721-29894535-jnxvl\" (UID: \"6ad15281-5d31-4d89-bd48-185887526864\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-b16721-29894535-jnxvl" Nov 03 02:15:00.307384 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:00.307331 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e1793d-29894535-ndjl5"] Nov 03 02:15:00.322571 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:00.322537 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fgwsl\"" Nov 03 02:15:00.362836 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:00.360459 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-98a394-29894535-x2v6t"] Nov 03 02:15:00.366401 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:00.365494 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/437aab4b-3661-46d1-a41f-8da45f894600-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-e1793d-29894535-ndjl5\" (UID: \"437aab4b-3661-46d1-a41f-8da45f894600\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e1793d-29894535-ndjl5" Nov 03 02:15:00.366401 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:00.365625 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d82m2\" (UniqueName: \"kubernetes.io/projected/437aab4b-3661-46d1-a41f-8da45f894600-kube-api-access-d82m2\") pod \"maas-api-key-cleanup-e2e-worker-w1-e1793d-29894535-ndjl5\" (UID: \"437aab4b-3661-46d1-a41f-8da45f894600\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e1793d-29894535-ndjl5" Nov 03 02:15:00.366401 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:00.365660 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa117337-4660-4c7b-ac91-a1a9d842bfc7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-98a394-29894535-x2v6t\" (UID: \"fa117337-4660-4c7b-ac91-a1a9d842bfc7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-98a394-29894535-x2v6t" Nov 03 02:15:00.366401 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:00.365714 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ad15281-5d31-4d89-bd48-185887526864-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-explicit-b16721-29894535-jnxvl\" (UID: \"6ad15281-5d31-4d89-bd48-185887526864\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-b16721-29894535-jnxvl" Nov 03 02:15:00.366401 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:00.365755 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6m8t\" (UniqueName: \"kubernetes.io/projected/fa117337-4660-4c7b-ac91-a1a9d842bfc7-kube-api-access-x6m8t\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-98a394-29894535-x2v6t\" (UID: \"fa117337-4660-4c7b-ac91-a1a9d842bfc7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-98a394-29894535-x2v6t" Nov 03 02:15:00.366401 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:00.365802 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7grrg\" (UniqueName: \"kubernetes.io/projected/6ad15281-5d31-4d89-bd48-185887526864-kube-api-access-7grrg\") pod \"maas-api-key-cleanup-e2e-explicit-b16721-29894535-jnxvl\" (UID: \"6ad15281-5d31-4d89-bd48-185887526864\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-b16721-29894535-jnxvl" Nov 03 02:15:00.368497 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:00.367025 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/437aab4b-3661-46d1-a41f-8da45f894600-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-e1793d-29894535-ndjl5\" (UID: \"437aab4b-3661-46d1-a41f-8da45f894600\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e1793d-29894535-ndjl5" Nov 03 02:15:00.368497 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:00.367978 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ad15281-5d31-4d89-bd48-185887526864-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-explicit-b16721-29894535-jnxvl\" (UID: \"6ad15281-5d31-4d89-bd48-185887526864\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-b16721-29894535-jnxvl" Nov 03 02:15:00.368497 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:00.368388 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-b16721-29894535-jnxvl"] Nov 03 02:15:00.429195 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:00.429163 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7grrg\" (UniqueName: \"kubernetes.io/projected/6ad15281-5d31-4d89-bd48-185887526864-kube-api-access-7grrg\") pod \"maas-api-key-cleanup-e2e-explicit-b16721-29894535-jnxvl\" (UID: \"6ad15281-5d31-4d89-bd48-185887526864\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-b16721-29894535-jnxvl" Nov 03 02:15:00.429680 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:00.429657 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d82m2\" (UniqueName: \"kubernetes.io/projected/437aab4b-3661-46d1-a41f-8da45f894600-kube-api-access-d82m2\") pod \"maas-api-key-cleanup-e2e-worker-w1-e1793d-29894535-ndjl5\" (UID: \"437aab4b-3661-46d1-a41f-8da45f894600\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e1793d-29894535-ndjl5" Nov 03 02:15:00.467739 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:00.467647 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa117337-4660-4c7b-ac91-a1a9d842bfc7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-98a394-29894535-x2v6t\" (UID: \"fa117337-4660-4c7b-ac91-a1a9d842bfc7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-98a394-29894535-x2v6t" Nov 03 02:15:00.467739 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:00.467711 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6m8t\" (UniqueName: \"kubernetes.io/projected/fa117337-4660-4c7b-ac91-a1a9d842bfc7-kube-api-access-x6m8t\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-98a394-29894535-x2v6t\" (UID: \"fa117337-4660-4c7b-ac91-a1a9d842bfc7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-98a394-29894535-x2v6t" Nov 03 02:15:00.468360 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:00.468336 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa117337-4660-4c7b-ac91-a1a9d842bfc7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-98a394-29894535-x2v6t\" (UID: \"fa117337-4660-4c7b-ac91-a1a9d842bfc7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-98a394-29894535-x2v6t" Nov 03 02:15:00.512305 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:00.512268 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e1793d-29894535-ndjl5" Nov 03 02:15:00.530781 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:00.530747 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6m8t\" (UniqueName: \"kubernetes.io/projected/fa117337-4660-4c7b-ac91-a1a9d842bfc7-kube-api-access-x6m8t\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-98a394-29894535-x2v6t\" (UID: \"fa117337-4660-4c7b-ac91-a1a9d842bfc7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-98a394-29894535-x2v6t" Nov 03 02:15:00.533644 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:00.533616 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-b16721-29894535-jnxvl" Nov 03 02:15:00.540948 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:00.540921 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-98a394-29894535-x2v6t" Nov 03 02:15:00.893181 ip-10-0-136-16 kubenswrapper[5437]: W1103 02:15:00.893149 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod437aab4b_3661_46d1_a41f_8da45f894600.slice/crio-345eec140e9c4c68d85a95de3cce8ed81411449138c07e77e61b8fdf7c28a279 WatchSource:0}: Error finding container 345eec140e9c4c68d85a95de3cce8ed81411449138c07e77e61b8fdf7c28a279: Status 404 returned error can't find the container with id 345eec140e9c4c68d85a95de3cce8ed81411449138c07e77e61b8fdf7c28a279 Nov 03 02:15:00.900208 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:00.896763 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e1793d-29894535-ndjl5"] Nov 03 02:15:00.938420 ip-10-0-136-16 kubenswrapper[5437]: W1103 02:15:00.938387 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa117337_4660_4c7b_ac91_a1a9d842bfc7.slice/crio-da9b81f044a13c0634685ce0b461e97b4d11f0491c10772d7df6c03aaeb6b1a7 WatchSource:0}: Error finding container da9b81f044a13c0634685ce0b461e97b4d11f0491c10772d7df6c03aaeb6b1a7: Status 404 returned error can't find the container with id da9b81f044a13c0634685ce0b461e97b4d11f0491c10772d7df6c03aaeb6b1a7 Nov 03 02:15:00.940480 ip-10-0-136-16 kubenswrapper[5437]: W1103 02:15:00.940453 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ad15281_5d31_4d89_bd48_185887526864.slice/crio-3ba15f22073582f48d731ca8a4fb794ab5d5486a49d0e08313b6c5936ea92ddd WatchSource:0}: Error finding container 3ba15f22073582f48d731ca8a4fb794ab5d5486a49d0e08313b6c5936ea92ddd: Status 404 returned error can't find the container with id 3ba15f22073582f48d731ca8a4fb794ab5d5486a49d0e08313b6c5936ea92ddd Nov 03 02:15:00.941610 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:00.941585 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-98a394-29894535-x2v6t"] Nov 03 02:15:00.944572 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:00.944545 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-b16721-29894535-jnxvl"] Nov 03 02:15:01.301186 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:01.301148 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-b16721-29894535-jnxvl" event={"ID":"6ad15281-5d31-4d89-bd48-185887526864","Type":"ContainerStarted","Data":"3ba15f22073582f48d731ca8a4fb794ab5d5486a49d0e08313b6c5936ea92ddd"} Nov 03 02:15:01.302603 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:01.302574 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-98a394-29894535-x2v6t" event={"ID":"fa117337-4660-4c7b-ac91-a1a9d842bfc7","Type":"ContainerStarted","Data":"da9b81f044a13c0634685ce0b461e97b4d11f0491c10772d7df6c03aaeb6b1a7"} Nov 03 02:15:01.303719 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:01.303697 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e1793d-29894535-ndjl5" event={"ID":"437aab4b-3661-46d1-a41f-8da45f894600","Type":"ContainerStarted","Data":"345eec140e9c4c68d85a95de3cce8ed81411449138c07e77e61b8fdf7c28a279"} Nov 03 02:15:01.818934 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:01.818877 5437 patch_prober.go:29] interesting pod/e2e-worker-w2-9da431-openshift-default-65c799fd77-kzj4s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 02:15:01.819149 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:01.818947 5437 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-kzj4s" podUID="ced2a84a-85cd-4435-bbca-33ee4ae19496" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 02:15:02.314024 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:02.313982 5437 generic.go:356] "Generic (PLEG): container finished" podID="6ad15281-5d31-4d89-bd48-185887526864" containerID="6a48486b0a305dfaf937075fb3b8440fbff2ba7cc70ce09700a27411a4910a96" exitCode=60 Nov 03 02:15:02.314582 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:02.314090 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-b16721-29894535-jnxvl" event={"ID":"6ad15281-5d31-4d89-bd48-185887526864","Type":"ContainerDied","Data":"6a48486b0a305dfaf937075fb3b8440fbff2ba7cc70ce09700a27411a4910a96"} Nov 03 02:15:02.314582 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:02.314424 5437 scope.go:122] "RemoveContainer" containerID="6a48486b0a305dfaf937075fb3b8440fbff2ba7cc70ce09700a27411a4910a96" Nov 03 02:15:02.316560 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:02.316537 5437 generic.go:356] "Generic (PLEG): container finished" podID="fa117337-4660-4c7b-ac91-a1a9d842bfc7" containerID="96c5e7eea75706ad77e9b893ccc7d1f9f604ef98c26f2add4e37b5102ac6f5e5" exitCode=60 Nov 03 02:15:02.316667 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:02.316572 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-98a394-29894535-x2v6t" event={"ID":"fa117337-4660-4c7b-ac91-a1a9d842bfc7","Type":"ContainerDied","Data":"96c5e7eea75706ad77e9b893ccc7d1f9f604ef98c26f2add4e37b5102ac6f5e5"} Nov 03 02:15:02.316895 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:02.316866 5437 scope.go:122] "RemoveContainer" containerID="96c5e7eea75706ad77e9b893ccc7d1f9f604ef98c26f2add4e37b5102ac6f5e5" Nov 03 02:15:02.318839 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:02.318818 5437 generic.go:356] "Generic (PLEG): container finished" podID="437aab4b-3661-46d1-a41f-8da45f894600" containerID="d086cd6fbf03af320d2b9cefa132707db8a9fd9be82158ac93a9463a7c8d20b1" exitCode=60 Nov 03 02:15:02.318946 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:02.318862 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e1793d-29894535-ndjl5" event={"ID":"437aab4b-3661-46d1-a41f-8da45f894600","Type":"ContainerDied","Data":"d086cd6fbf03af320d2b9cefa132707db8a9fd9be82158ac93a9463a7c8d20b1"} Nov 03 02:15:02.319175 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:02.319159 5437 scope.go:122] "RemoveContainer" containerID="d086cd6fbf03af320d2b9cefa132707db8a9fd9be82158ac93a9463a7c8d20b1" Nov 03 02:15:02.626757 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:02.626717 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:15:03.250430 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:03.250399 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c70365-58bfdd8b7f-qw6f2" Nov 03 02:15:03.328635 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:03.328596 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-98a394-29894535-x2v6t" event={"ID":"fa117337-4660-4c7b-ac91-a1a9d842bfc7","Type":"ContainerStarted","Data":"2850d2185a4d382b75a12949074d5e75184be53dbca8425c0d05f4109f136a72"} Nov 03 02:15:03.330704 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:03.330675 5437 generic.go:356] "Generic (PLEG): container finished" podID="437aab4b-3661-46d1-a41f-8da45f894600" containerID="339f766af60a4f843ade55444497aaa2fb5f7e3beaea8a1a8bd19dba94dd30ed" exitCode=60 Nov 03 02:15:03.330975 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:03.330723 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e1793d-29894535-ndjl5" event={"ID":"437aab4b-3661-46d1-a41f-8da45f894600","Type":"ContainerDied","Data":"339f766af60a4f843ade55444497aaa2fb5f7e3beaea8a1a8bd19dba94dd30ed"} Nov 03 02:15:03.330975 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:03.330793 5437 scope.go:122] "RemoveContainer" containerID="d086cd6fbf03af320d2b9cefa132707db8a9fd9be82158ac93a9463a7c8d20b1" Nov 03 02:15:03.331093 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:03.331053 5437 scope.go:122] "RemoveContainer" containerID="339f766af60a4f843ade55444497aaa2fb5f7e3beaea8a1a8bd19dba94dd30ed" Nov 03 02:15:03.331445 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:15:03.331421 5437 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-e1793d-29894535-ndjl5_odh-ai-gateway-infra(437aab4b-3661-46d1-a41f-8da45f894600)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e1793d-29894535-ndjl5" podUID="437aab4b-3661-46d1-a41f-8da45f894600" Nov 03 02:15:03.333072 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:03.333045 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-b16721-29894535-jnxvl" event={"ID":"6ad15281-5d31-4d89-bd48-185887526864","Type":"ContainerStarted","Data":"db7891c3fb0ebe61152b8a5f98358353c1ae27082f87fe15e99b517418a8d02a"} Nov 03 02:15:03.334946 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:03.334922 5437 generic.go:356] "Generic (PLEG): container finished" podID="ced2a84a-85cd-4435-bbca-33ee4ae19496" containerID="ba41559d0eeaf4d3a5557e8b0700d53f0c491d8b40c1fafeba2241cb14129d1d" exitCode=0 Nov 03 02:15:03.335051 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:03.334987 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-kzj4s" event={"ID":"ced2a84a-85cd-4435-bbca-33ee4ae19496","Type":"ContainerDied","Data":"ba41559d0eeaf4d3a5557e8b0700d53f0c491d8b40c1fafeba2241cb14129d1d"} Nov 03 02:15:03.562210 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:03.562180 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-kzj4s" Nov 03 02:15:03.608852 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:03.608790 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-b16721-29894535-jnxvl" podStartSLOduration=3.608774358 podStartE2EDuration="3.608774358s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:15:03.606049322 +0000 UTC m=+3385.635531616" watchObservedRunningTime="2026-11-03 02:15:03.608774358 +0000 UTC m=+3385.638256654" Nov 03 02:15:03.708765 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:03.708669 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ced2a84a-85cd-4435-bbca-33ee4ae19496-istio-token\" (UniqueName: \"kubernetes.io/projected/ced2a84a-85cd-4435-bbca-33ee4ae19496-istio-token\") pod \"ced2a84a-85cd-4435-bbca-33ee4ae19496\" (UID: \"ced2a84a-85cd-4435-bbca-33ee4ae19496\") " Nov 03 02:15:03.708765 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:03.708759 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ced2a84a-85cd-4435-bbca-33ee4ae19496-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ced2a84a-85cd-4435-bbca-33ee4ae19496-istio-podinfo\") pod \"ced2a84a-85cd-4435-bbca-33ee4ae19496\" (UID: \"ced2a84a-85cd-4435-bbca-33ee4ae19496\") " Nov 03 02:15:03.709004 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:03.708818 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ced2a84a-85cd-4435-bbca-33ee4ae19496-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ced2a84a-85cd-4435-bbca-33ee4ae19496-istio-data\") pod \"ced2a84a-85cd-4435-bbca-33ee4ae19496\" (UID: \"ced2a84a-85cd-4435-bbca-33ee4ae19496\") " Nov 03 02:15:03.709004 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:03.708845 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ced2a84a-85cd-4435-bbca-33ee4ae19496-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ced2a84a-85cd-4435-bbca-33ee4ae19496-istiod-ca-cert\") pod \"ced2a84a-85cd-4435-bbca-33ee4ae19496\" (UID: \"ced2a84a-85cd-4435-bbca-33ee4ae19496\") " Nov 03 02:15:03.709004 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:03.708878 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ced2a84a-85cd-4435-bbca-33ee4ae19496-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ced2a84a-85cd-4435-bbca-33ee4ae19496-istio-envoy\") pod \"ced2a84a-85cd-4435-bbca-33ee4ae19496\" (UID: \"ced2a84a-85cd-4435-bbca-33ee4ae19496\") " Nov 03 02:15:03.709004 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:03.708968 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ced2a84a-85cd-4435-bbca-33ee4ae19496-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ced2a84a-85cd-4435-bbca-33ee4ae19496-credential-socket\") pod \"ced2a84a-85cd-4435-bbca-33ee4ae19496\" (UID: \"ced2a84a-85cd-4435-bbca-33ee4ae19496\") " Nov 03 02:15:03.709217 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:03.709086 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ced2a84a-85cd-4435-bbca-33ee4ae19496-kube-api-access-2n89j\" (UniqueName: \"kubernetes.io/projected/ced2a84a-85cd-4435-bbca-33ee4ae19496-kube-api-access-2n89j\") pod \"ced2a84a-85cd-4435-bbca-33ee4ae19496\" (UID: \"ced2a84a-85cd-4435-bbca-33ee4ae19496\") " Nov 03 02:15:03.709217 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:03.709141 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ced2a84a-85cd-4435-bbca-33ee4ae19496-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ced2a84a-85cd-4435-bbca-33ee4ae19496-workload-certs\") pod \"ced2a84a-85cd-4435-bbca-33ee4ae19496\" (UID: \"ced2a84a-85cd-4435-bbca-33ee4ae19496\") " Nov 03 02:15:03.709217 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:03.709147 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ced2a84a-85cd-4435-bbca-33ee4ae19496-istio-data" pod "ced2a84a-85cd-4435-bbca-33ee4ae19496" (UID: "ced2a84a-85cd-4435-bbca-33ee4ae19496"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:15:03.709217 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:03.709204 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ced2a84a-85cd-4435-bbca-33ee4ae19496-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ced2a84a-85cd-4435-bbca-33ee4ae19496-workload-socket\") pod \"ced2a84a-85cd-4435-bbca-33ee4ae19496\" (UID: \"ced2a84a-85cd-4435-bbca-33ee4ae19496\") " Nov 03 02:15:03.709447 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:03.709254 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ced2a84a-85cd-4435-bbca-33ee4ae19496-credential-socket" pod "ced2a84a-85cd-4435-bbca-33ee4ae19496" (UID: "ced2a84a-85cd-4435-bbca-33ee4ae19496"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:15:03.709447 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:03.709288 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ced2a84a-85cd-4435-bbca-33ee4ae19496-istiod-ca-cert" pod "ced2a84a-85cd-4435-bbca-33ee4ae19496" (UID: "ced2a84a-85cd-4435-bbca-33ee4ae19496"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:15:03.709545 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:03.709504 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ced2a84a-85cd-4435-bbca-33ee4ae19496-workload-certs" pod "ced2a84a-85cd-4435-bbca-33ee4ae19496" (UID: "ced2a84a-85cd-4435-bbca-33ee4ae19496"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:15:03.709684 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:03.709655 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ced2a84a-85cd-4435-bbca-33ee4ae19496-workload-socket" pod "ced2a84a-85cd-4435-bbca-33ee4ae19496" (UID: "ced2a84a-85cd-4435-bbca-33ee4ae19496"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:15:03.709684 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:03.709688 5437 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ced2a84a-85cd-4435-bbca-33ee4ae19496-workload-certs\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:15:03.709894 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:03.709707 5437 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ced2a84a-85cd-4435-bbca-33ee4ae19496-istio-data\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:15:03.709894 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:03.709718 5437 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ced2a84a-85cd-4435-bbca-33ee4ae19496-istiod-ca-cert\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:15:03.709894 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:03.709727 5437 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ced2a84a-85cd-4435-bbca-33ee4ae19496-credential-socket\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:15:03.711266 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:03.711244 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ced2a84a-85cd-4435-bbca-33ee4ae19496-kube-api-access-2n89j" pod "ced2a84a-85cd-4435-bbca-33ee4ae19496" (UID: "ced2a84a-85cd-4435-bbca-33ee4ae19496"). InnerVolumeSpecName "kube-api-access-2n89j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:15:03.711385 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:03.711277 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ced2a84a-85cd-4435-bbca-33ee4ae19496-istio-token" pod "ced2a84a-85cd-4435-bbca-33ee4ae19496" (UID: "ced2a84a-85cd-4435-bbca-33ee4ae19496"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:15:03.711525 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:03.711493 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ced2a84a-85cd-4435-bbca-33ee4ae19496-istio-envoy" pod "ced2a84a-85cd-4435-bbca-33ee4ae19496" (UID: "ced2a84a-85cd-4435-bbca-33ee4ae19496"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:15:03.711566 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:03.711512 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ced2a84a-85cd-4435-bbca-33ee4ae19496-istio-podinfo" pod "ced2a84a-85cd-4435-bbca-33ee4ae19496" (UID: "ced2a84a-85cd-4435-bbca-33ee4ae19496"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:15:03.724393 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:03.724323 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-98a394-29894535-x2v6t" podStartSLOduration=3.724302511 podStartE2EDuration="3.724302511s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:15:03.723998078 +0000 UTC m=+3385.753480373" watchObservedRunningTime="2026-11-03 02:15:03.724302511 +0000 UTC m=+3385.753784807" Nov 03 02:15:03.811267 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:03.811229 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2n89j\" (UniqueName: \"kubernetes.io/projected/ced2a84a-85cd-4435-bbca-33ee4ae19496-kube-api-access-2n89j\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:15:03.811267 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:03.811266 5437 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ced2a84a-85cd-4435-bbca-33ee4ae19496-workload-socket\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:15:03.811267 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:03.811278 5437 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ced2a84a-85cd-4435-bbca-33ee4ae19496-istio-token\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:15:03.811541 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:03.811287 5437 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ced2a84a-85cd-4435-bbca-33ee4ae19496-istio-podinfo\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:15:03.811541 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:03.811296 5437 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ced2a84a-85cd-4435-bbca-33ee4ae19496-istio-envoy\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:15:04.165323 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:04.165272 5437 pod_container_manager_linux.go:218] "Failed to delete cgroup paths" cgroupName=["kubepods","burstable","podb2740d82-3082-474a-8726-4aaaecf57272"] err="unable to destroy cgroup paths for cgroup [kubepods burstable podb2740d82-3082-474a-8726-4aaaecf57272] : Timed out while waiting for systemd to remove kubepods-burstable-podb2740d82_3082_474a_8726_4aaaecf57272.slice" Nov 03 02:15:04.165323 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:04.165323 5437 kubelet_pods.go:2797] "Failed to destroy orphaned pod cgroup" podUID="b2740d82-3082-474a-8726-4aaaecf57272" err="failed to delete cgroup paths for [kubepods burstable podb2740d82-3082-474a-8726-4aaaecf57272] : unable to destroy cgroup paths for cgroup [kubepods burstable podb2740d82-3082-474a-8726-4aaaecf57272] : Timed out while waiting for systemd to remove kubepods-burstable-podb2740d82_3082_474a_8726_4aaaecf57272.slice" Nov 03 02:15:04.344479 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:04.344441 5437 generic.go:356] "Generic (PLEG): container finished" podID="6ad15281-5d31-4d89-bd48-185887526864" containerID="db7891c3fb0ebe61152b8a5f98358353c1ae27082f87fe15e99b517418a8d02a" exitCode=60 Nov 03 02:15:04.344967 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:04.344528 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-b16721-29894535-jnxvl" event={"ID":"6ad15281-5d31-4d89-bd48-185887526864","Type":"ContainerDied","Data":"db7891c3fb0ebe61152b8a5f98358353c1ae27082f87fe15e99b517418a8d02a"} Nov 03 02:15:04.344967 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:04.344597 5437 scope.go:122] "RemoveContainer" containerID="6a48486b0a305dfaf937075fb3b8440fbff2ba7cc70ce09700a27411a4910a96" Nov 03 02:15:04.344967 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:04.344897 5437 scope.go:122] "RemoveContainer" containerID="db7891c3fb0ebe61152b8a5f98358353c1ae27082f87fe15e99b517418a8d02a" Nov 03 02:15:04.345278 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:15:04.345247 5437 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-explicit-b16721-29894535-jnxvl_odh-ai-gateway-infra(6ad15281-5d31-4d89-bd48-185887526864)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-b16721-29894535-jnxvl" podUID="6ad15281-5d31-4d89-bd48-185887526864" Nov 03 02:15:04.346099 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:04.346080 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-kzj4s" Nov 03 02:15:04.346180 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:04.346105 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-kzj4s" event={"ID":"ced2a84a-85cd-4435-bbca-33ee4ae19496","Type":"ContainerDied","Data":"02171d783cae11aefcf8460cb2df79a89905d45cc7cdd5b079e0fd23955a4b03"} Nov 03 02:15:04.348550 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:04.348465 5437 generic.go:356] "Generic (PLEG): container finished" podID="fa117337-4660-4c7b-ac91-a1a9d842bfc7" containerID="2850d2185a4d382b75a12949074d5e75184be53dbca8425c0d05f4109f136a72" exitCode=60 Nov 03 02:15:04.348550 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:04.348546 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-98a394-29894535-x2v6t" event={"ID":"fa117337-4660-4c7b-ac91-a1a9d842bfc7","Type":"ContainerDied","Data":"2850d2185a4d382b75a12949074d5e75184be53dbca8425c0d05f4109f136a72"} Nov 03 02:15:04.348875 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:04.348855 5437 scope.go:122] "RemoveContainer" containerID="2850d2185a4d382b75a12949074d5e75184be53dbca8425c0d05f4109f136a72" Nov 03 02:15:04.349149 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:15:04.349131 5437 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-98a394-29894535-x2v6t_odh-ai-gateway-infra(fa117337-4660-4c7b-ac91-a1a9d842bfc7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-98a394-29894535-x2v6t" podUID="fa117337-4660-4c7b-ac91-a1a9d842bfc7" Nov 03 02:15:04.352304 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:04.352279 5437 scope.go:122] "RemoveContainer" containerID="339f766af60a4f843ade55444497aaa2fb5f7e3beaea8a1a8bd19dba94dd30ed" Nov 03 02:15:04.352646 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:15:04.352619 5437 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-e1793d-29894535-ndjl5_odh-ai-gateway-infra(437aab4b-3661-46d1-a41f-8da45f894600)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e1793d-29894535-ndjl5" podUID="437aab4b-3661-46d1-a41f-8da45f894600" Nov 03 02:15:04.359271 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:04.359246 5437 scope.go:122] "RemoveContainer" containerID="ba41559d0eeaf4d3a5557e8b0700d53f0c491d8b40c1fafeba2241cb14129d1d" Nov 03 02:15:04.369783 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:04.369757 5437 scope.go:122] "RemoveContainer" containerID="96c5e7eea75706ad77e9b893ccc7d1f9f604ef98c26f2add4e37b5102ac6f5e5" Nov 03 02:15:04.700084 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:04.699994 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-kzj4s"] Nov 03 02:15:04.703972 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:04.703941 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-kzj4s"] Nov 03 02:15:05.362032 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:05.362001 5437 scope.go:122] "RemoveContainer" containerID="2850d2185a4d382b75a12949074d5e75184be53dbca8425c0d05f4109f136a72" Nov 03 02:15:05.362486 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:15:05.362334 5437 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-98a394-29894535-x2v6t_odh-ai-gateway-infra(fa117337-4660-4c7b-ac91-a1a9d842bfc7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-98a394-29894535-x2v6t" podUID="fa117337-4660-4c7b-ac91-a1a9d842bfc7" Nov 03 02:15:05.363859 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:05.363838 5437 scope.go:122] "RemoveContainer" containerID="db7891c3fb0ebe61152b8a5f98358353c1ae27082f87fe15e99b517418a8d02a" Nov 03 02:15:05.364171 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:15:05.364143 5437 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-explicit-b16721-29894535-jnxvl_odh-ai-gateway-infra(6ad15281-5d31-4d89-bd48-185887526864)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-b16721-29894535-jnxvl" podUID="6ad15281-5d31-4d89-bd48-185887526864" Nov 03 02:15:06.090138 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:06.090101 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ced2a84a-85cd-4435-bbca-33ee4ae19496" path="/var/lib/kubelet/pods/ced2a84a-85cd-4435-bbca-33ee4ae19496/volumes" Nov 03 02:15:07.115625 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:07.115577 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:15:11.867536 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:11.867490 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:15:14.036404 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:15:14.036329 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 02:15:16.082325 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:16.082290 5437 scope.go:122] "RemoveContainer" containerID="db7891c3fb0ebe61152b8a5f98358353c1ae27082f87fe15e99b517418a8d02a" Nov 03 02:15:16.439945 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:16.439911 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-b16721-29894535-jnxvl" event={"ID":"6ad15281-5d31-4d89-bd48-185887526864","Type":"ContainerStarted","Data":"95422c0313bf6d2d0b0a66e240836fe1e78fca766760129f58e9953985846f50"} Nov 03 02:15:16.697869 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:16.697772 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:15:17.082169 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:17.082128 5437 scope.go:122] "RemoveContainer" containerID="339f766af60a4f843ade55444497aaa2fb5f7e3beaea8a1a8bd19dba94dd30ed" Nov 03 02:15:17.082414 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:17.082216 5437 scope.go:122] "RemoveContainer" containerID="2850d2185a4d382b75a12949074d5e75184be53dbca8425c0d05f4109f136a72" Nov 03 02:15:17.448912 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:17.448880 5437 generic.go:356] "Generic (PLEG): container finished" podID="6ad15281-5d31-4d89-bd48-185887526864" containerID="95422c0313bf6d2d0b0a66e240836fe1e78fca766760129f58e9953985846f50" exitCode=60 Nov 03 02:15:17.449098 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:17.448956 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-b16721-29894535-jnxvl" event={"ID":"6ad15281-5d31-4d89-bd48-185887526864","Type":"ContainerDied","Data":"95422c0313bf6d2d0b0a66e240836fe1e78fca766760129f58e9953985846f50"} Nov 03 02:15:17.449098 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:17.449041 5437 scope.go:122] "RemoveContainer" containerID="db7891c3fb0ebe61152b8a5f98358353c1ae27082f87fe15e99b517418a8d02a" Nov 03 02:15:17.449520 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:17.449497 5437 scope.go:122] "RemoveContainer" containerID="95422c0313bf6d2d0b0a66e240836fe1e78fca766760129f58e9953985846f50" Nov 03 02:15:17.449838 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:15:17.449818 5437 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-explicit-b16721-29894535-jnxvl_odh-ai-gateway-infra(6ad15281-5d31-4d89-bd48-185887526864)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-b16721-29894535-jnxvl" podUID="6ad15281-5d31-4d89-bd48-185887526864" Nov 03 02:15:17.451208 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:17.451174 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-98a394-29894535-x2v6t" event={"ID":"fa117337-4660-4c7b-ac91-a1a9d842bfc7","Type":"ContainerStarted","Data":"130601c2474bb00531ed529a6174620af9fb18b700bd2739b33c88cee2091f12"} Nov 03 02:15:17.453968 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:17.453940 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e1793d-29894535-ndjl5" event={"ID":"437aab4b-3661-46d1-a41f-8da45f894600","Type":"ContainerStarted","Data":"d4b6599dc5b37ca847896cdfc0b9abb924772010736b03c8f17df04d8b63176a"} Nov 03 02:15:17.575077 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:17.575040 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-b16721-29894535-jnxvl"] Nov 03 02:15:17.824582 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:17.824532 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e1793d-29894535-ndjl5" podStartSLOduration=17.82451763 podStartE2EDuration="17.82451763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:15:17.82392619 +0000 UTC m=+3399.853408486" watchObservedRunningTime="2026-11-03 02:15:17.82451763 +0000 UTC m=+3399.853999925" Nov 03 02:15:18.462405 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:18.462338 5437 generic.go:356] "Generic (PLEG): container finished" podID="fa117337-4660-4c7b-ac91-a1a9d842bfc7" containerID="130601c2474bb00531ed529a6174620af9fb18b700bd2739b33c88cee2091f12" exitCode=60 Nov 03 02:15:18.462990 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:18.462406 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-98a394-29894535-x2v6t" event={"ID":"fa117337-4660-4c7b-ac91-a1a9d842bfc7","Type":"ContainerDied","Data":"130601c2474bb00531ed529a6174620af9fb18b700bd2739b33c88cee2091f12"} Nov 03 02:15:18.462990 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:18.462462 5437 scope.go:122] "RemoveContainer" containerID="2850d2185a4d382b75a12949074d5e75184be53dbca8425c0d05f4109f136a72" Nov 03 02:15:18.462990 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:18.462848 5437 scope.go:122] "RemoveContainer" containerID="130601c2474bb00531ed529a6174620af9fb18b700bd2739b33c88cee2091f12" Nov 03 02:15:18.463193 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:15:18.463172 5437 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-98a394-29894535-x2v6t_odh-ai-gateway-infra(fa117337-4660-4c7b-ac91-a1a9d842bfc7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-98a394-29894535-x2v6t" podUID="fa117337-4660-4c7b-ac91-a1a9d842bfc7" Nov 03 02:15:18.464790 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:18.464766 5437 generic.go:356] "Generic (PLEG): container finished" podID="437aab4b-3661-46d1-a41f-8da45f894600" containerID="d4b6599dc5b37ca847896cdfc0b9abb924772010736b03c8f17df04d8b63176a" exitCode=60 Nov 03 02:15:18.464889 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:18.464846 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e1793d-29894535-ndjl5" event={"ID":"437aab4b-3661-46d1-a41f-8da45f894600","Type":"ContainerDied","Data":"d4b6599dc5b37ca847896cdfc0b9abb924772010736b03c8f17df04d8b63176a"} Nov 03 02:15:18.465263 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:18.465244 5437 scope.go:122] "RemoveContainer" containerID="d4b6599dc5b37ca847896cdfc0b9abb924772010736b03c8f17df04d8b63176a" Nov 03 02:15:18.465594 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:15:18.465567 5437 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-e1793d-29894535-ndjl5_odh-ai-gateway-infra(437aab4b-3661-46d1-a41f-8da45f894600)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e1793d-29894535-ndjl5" podUID="437aab4b-3661-46d1-a41f-8da45f894600" Nov 03 02:15:18.478128 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:18.478105 5437 scope.go:122] "RemoveContainer" containerID="339f766af60a4f843ade55444497aaa2fb5f7e3beaea8a1a8bd19dba94dd30ed" Nov 03 02:15:18.618952 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:18.618927 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-b16721-29894535-jnxvl" Nov 03 02:15:18.696969 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:18.696937 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ad15281-5d31-4d89-bd48-185887526864-kube-api-access-7grrg\" (UniqueName: \"kubernetes.io/projected/6ad15281-5d31-4d89-bd48-185887526864-kube-api-access-7grrg\") pod \"6ad15281-5d31-4d89-bd48-185887526864\" (UID: \"6ad15281-5d31-4d89-bd48-185887526864\") " Nov 03 02:15:18.697155 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:18.697059 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ad15281-5d31-4d89-bd48-185887526864-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ad15281-5d31-4d89-bd48-185887526864-maas-api-service-ca\") pod \"6ad15281-5d31-4d89-bd48-185887526864\" (UID: \"6ad15281-5d31-4d89-bd48-185887526864\") " Nov 03 02:15:18.697505 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:18.697473 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ad15281-5d31-4d89-bd48-185887526864-maas-api-service-ca" pod "6ad15281-5d31-4d89-bd48-185887526864" (UID: "6ad15281-5d31-4d89-bd48-185887526864"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:15:18.699299 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:18.699275 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ad15281-5d31-4d89-bd48-185887526864-kube-api-access-7grrg" pod "6ad15281-5d31-4d89-bd48-185887526864" (UID: "6ad15281-5d31-4d89-bd48-185887526864"). InnerVolumeSpecName "kube-api-access-7grrg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:15:18.714193 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:18.714129 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-98a394-29894535-x2v6t"] Nov 03 02:15:18.798707 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:18.798674 5437 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ad15281-5d31-4d89-bd48-185887526864-maas-api-service-ca\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:15:18.798707 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:18.798703 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7grrg\" (UniqueName: \"kubernetes.io/projected/6ad15281-5d31-4d89-bd48-185887526864-kube-api-access-7grrg\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:15:18.834119 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:18.834080 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e1793d-29894535-ndjl5"] Nov 03 02:15:19.479810 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:19.479787 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-b16721-29894535-jnxvl" Nov 03 02:15:19.480213 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:19.479821 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-b16721-29894535-jnxvl" event={"ID":"6ad15281-5d31-4d89-bd48-185887526864","Type":"ContainerDied","Data":"3ba15f22073582f48d731ca8a4fb794ab5d5486a49d0e08313b6c5936ea92ddd"} Nov 03 02:15:19.480213 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:19.479886 5437 scope.go:122] "RemoveContainer" containerID="95422c0313bf6d2d0b0a66e240836fe1e78fca766760129f58e9953985846f50" Nov 03 02:15:19.681842 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:19.681812 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-98a394-29894535-x2v6t" Nov 03 02:15:19.685286 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:19.685261 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e1793d-29894535-ndjl5" Nov 03 02:15:19.717519 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:19.717486 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa117337-4660-4c7b-ac91-a1a9d842bfc7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa117337-4660-4c7b-ac91-a1a9d842bfc7-maas-api-service-ca\") pod \"fa117337-4660-4c7b-ac91-a1a9d842bfc7\" (UID: \"fa117337-4660-4c7b-ac91-a1a9d842bfc7\") " Nov 03 02:15:19.717696 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:19.717548 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa117337-4660-4c7b-ac91-a1a9d842bfc7-kube-api-access-x6m8t\" (UniqueName: \"kubernetes.io/projected/fa117337-4660-4c7b-ac91-a1a9d842bfc7-kube-api-access-x6m8t\") pod \"fa117337-4660-4c7b-ac91-a1a9d842bfc7\" (UID: \"fa117337-4660-4c7b-ac91-a1a9d842bfc7\") " Nov 03 02:15:19.717949 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:19.717922 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa117337-4660-4c7b-ac91-a1a9d842bfc7-maas-api-service-ca" pod "fa117337-4660-4c7b-ac91-a1a9d842bfc7" (UID: "fa117337-4660-4c7b-ac91-a1a9d842bfc7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:15:19.719827 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:19.719796 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa117337-4660-4c7b-ac91-a1a9d842bfc7-kube-api-access-x6m8t" pod "fa117337-4660-4c7b-ac91-a1a9d842bfc7" (UID: "fa117337-4660-4c7b-ac91-a1a9d842bfc7"). InnerVolumeSpecName "kube-api-access-x6m8t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:15:19.818975 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:19.818934 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/437aab4b-3661-46d1-a41f-8da45f894600-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/437aab4b-3661-46d1-a41f-8da45f894600-maas-api-service-ca\") pod \"437aab4b-3661-46d1-a41f-8da45f894600\" (UID: \"437aab4b-3661-46d1-a41f-8da45f894600\") " Nov 03 02:15:19.819226 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:19.819208 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/437aab4b-3661-46d1-a41f-8da45f894600-kube-api-access-d82m2\" (UniqueName: \"kubernetes.io/projected/437aab4b-3661-46d1-a41f-8da45f894600-kube-api-access-d82m2\") pod \"437aab4b-3661-46d1-a41f-8da45f894600\" (UID: \"437aab4b-3661-46d1-a41f-8da45f894600\") " Nov 03 02:15:19.819328 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:19.819307 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/437aab4b-3661-46d1-a41f-8da45f894600-maas-api-service-ca" pod "437aab4b-3661-46d1-a41f-8da45f894600" (UID: "437aab4b-3661-46d1-a41f-8da45f894600"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:15:19.819622 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:19.819599 5437 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa117337-4660-4c7b-ac91-a1a9d842bfc7-maas-api-service-ca\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:15:19.819622 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:19.819621 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6m8t\" (UniqueName: \"kubernetes.io/projected/fa117337-4660-4c7b-ac91-a1a9d842bfc7-kube-api-access-x6m8t\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:15:19.819797 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:19.819637 5437 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/437aab4b-3661-46d1-a41f-8da45f894600-maas-api-service-ca\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:15:19.821377 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:19.821343 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/437aab4b-3661-46d1-a41f-8da45f894600-kube-api-access-d82m2" pod "437aab4b-3661-46d1-a41f-8da45f894600" (UID: "437aab4b-3661-46d1-a41f-8da45f894600"). InnerVolumeSpecName "kube-api-access-d82m2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:15:19.833262 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:19.833224 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-b16721-29894535-jnxvl"] Nov 03 02:15:19.835592 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:19.835570 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-b16721-29894535-jnxvl"] Nov 03 02:15:19.921098 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:19.921055 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d82m2\" (UniqueName: \"kubernetes.io/projected/437aab4b-3661-46d1-a41f-8da45f894600-kube-api-access-d82m2\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:15:20.098738 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:20.098697 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ad15281-5d31-4d89-bd48-185887526864" path="/var/lib/kubelet/pods/6ad15281-5d31-4d89-bd48-185887526864/volumes" Nov 03 02:15:20.489208 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:20.489116 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-98a394-29894535-x2v6t" Nov 03 02:15:20.489724 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:20.489115 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-98a394-29894535-x2v6t" event={"ID":"fa117337-4660-4c7b-ac91-a1a9d842bfc7","Type":"ContainerDied","Data":"da9b81f044a13c0634685ce0b461e97b4d11f0491c10772d7df6c03aaeb6b1a7"} Nov 03 02:15:20.489724 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:20.489291 5437 scope.go:122] "RemoveContainer" containerID="130601c2474bb00531ed529a6174620af9fb18b700bd2739b33c88cee2091f12" Nov 03 02:15:20.490740 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:20.490708 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e1793d-29894535-ndjl5" Nov 03 02:15:20.490740 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:20.490711 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e1793d-29894535-ndjl5" event={"ID":"437aab4b-3661-46d1-a41f-8da45f894600","Type":"ContainerDied","Data":"345eec140e9c4c68d85a95de3cce8ed81411449138c07e77e61b8fdf7c28a279"} Nov 03 02:15:20.499653 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:20.499633 5437 scope.go:122] "RemoveContainer" containerID="d4b6599dc5b37ca847896cdfc0b9abb924772010736b03c8f17df04d8b63176a" Nov 03 02:15:20.618837 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:20.618797 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7014-8c2f98"] Nov 03 02:15:20.619391 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:20.619359 5437 cpu_manager.go:414] "containerMap: removing container" podUID="437aab4b-3661-46d1-a41f-8da45f894600" containerName="cleanup" Nov 03 02:15:20.619391 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:20.619388 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="437aab4b-3661-46d1-a41f-8da45f894600" containerName="cleanup" Nov 03 02:15:20.619473 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:20.619400 5437 cpu_manager.go:414] "containerMap: removing container" podUID="437aab4b-3661-46d1-a41f-8da45f894600" containerName="cleanup" Nov 03 02:15:20.619473 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:20.619405 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="437aab4b-3661-46d1-a41f-8da45f894600" containerName="cleanup" Nov 03 02:15:20.619473 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:20.619417 5437 cpu_manager.go:414] "containerMap: removing container" podUID="fa117337-4660-4c7b-ac91-a1a9d842bfc7" containerName="cleanup" Nov 03 02:15:20.619473 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:20.619423 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa117337-4660-4c7b-ac91-a1a9d842bfc7" containerName="cleanup" Nov 03 02:15:20.619473 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:20.619430 5437 cpu_manager.go:414] "containerMap: removing container" podUID="6ad15281-5d31-4d89-bd48-185887526864" containerName="cleanup" Nov 03 02:15:20.619473 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:20.619435 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ad15281-5d31-4d89-bd48-185887526864" containerName="cleanup" Nov 03 02:15:20.619473 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:20.619445 5437 cpu_manager.go:414] "containerMap: removing container" podUID="ced2a84a-85cd-4435-bbca-33ee4ae19496" containerName="istio-proxy" Nov 03 02:15:20.619473 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:20.619449 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ced2a84a-85cd-4435-bbca-33ee4ae19496" containerName="istio-proxy" Nov 03 02:15:20.619473 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:20.619456 5437 cpu_manager.go:414] "containerMap: removing container" podUID="6ad15281-5d31-4d89-bd48-185887526864" containerName="cleanup" Nov 03 02:15:20.619473 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:20.619461 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ad15281-5d31-4d89-bd48-185887526864" containerName="cleanup" Nov 03 02:15:20.619473 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:20.619473 5437 cpu_manager.go:414] "containerMap: removing container" podUID="6ad15281-5d31-4d89-bd48-185887526864" containerName="cleanup" Nov 03 02:15:20.619818 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:20.619478 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ad15281-5d31-4d89-bd48-185887526864" containerName="cleanup" Nov 03 02:15:20.619818 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:20.619490 5437 cpu_manager.go:414] "containerMap: removing container" podUID="fa117337-4660-4c7b-ac91-a1a9d842bfc7" containerName="cleanup" Nov 03 02:15:20.619818 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:20.619494 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa117337-4660-4c7b-ac91-a1a9d842bfc7" containerName="cleanup" Nov 03 02:15:20.619818 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:20.619602 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="437aab4b-3661-46d1-a41f-8da45f894600" containerName="cleanup" Nov 03 02:15:20.619818 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:20.619612 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="437aab4b-3661-46d1-a41f-8da45f894600" containerName="cleanup" Nov 03 02:15:20.619818 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:20.619619 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa117337-4660-4c7b-ac91-a1a9d842bfc7" containerName="cleanup" Nov 03 02:15:20.619818 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:20.619623 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ad15281-5d31-4d89-bd48-185887526864" containerName="cleanup" Nov 03 02:15:20.619818 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:20.619629 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="ced2a84a-85cd-4435-bbca-33ee4ae19496" containerName="istio-proxy" Nov 03 02:15:20.619818 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:20.619636 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ad15281-5d31-4d89-bd48-185887526864" containerName="cleanup" Nov 03 02:15:20.619818 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:20.619643 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa117337-4660-4c7b-ac91-a1a9d842bfc7" containerName="cleanup" Nov 03 02:15:20.619818 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:20.619649 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa117337-4660-4c7b-ac91-a1a9d842bfc7" containerName="cleanup" Nov 03 02:15:20.624804 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:20.624780 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7014-8c2f98" Nov 03 02:15:20.627867 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:20.627831 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:15:20.661510 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:20.661475 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e1793d-29894535-ndjl5"] Nov 03 02:15:20.673452 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:20.673422 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e1793d-29894535-ndjl5"] Nov 03 02:15:20.673960 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:20.673940 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7014-8c2f98"] Nov 03 02:15:20.710529 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:20.710498 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-882bv\"" Nov 03 02:15:20.731294 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:20.731258 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w86sw\" (UniqueName: \"kubernetes.io/projected/e10dcd70-fd32-4d44-8837-ae13ea3bea30-kube-api-access-w86sw\") pod \"test-curl-7014-8c2f98\" (UID: \"e10dcd70-fd32-4d44-8837-ae13ea3bea30\") " pod="opendatahub/test-curl-7014-8c2f98" Nov 03 02:15:20.778922 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:20.778839 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-98a394-29894535-x2v6t"] Nov 03 02:15:20.783686 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:20.783656 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-98a394-29894535-x2v6t"] Nov 03 02:15:20.832802 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:20.832765 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w86sw\" (UniqueName: \"kubernetes.io/projected/e10dcd70-fd32-4d44-8837-ae13ea3bea30-kube-api-access-w86sw\") pod \"test-curl-7014-8c2f98\" (UID: \"e10dcd70-fd32-4d44-8837-ae13ea3bea30\") " pod="opendatahub/test-curl-7014-8c2f98" Nov 03 02:15:20.895909 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:20.895872 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w86sw\" (UniqueName: \"kubernetes.io/projected/e10dcd70-fd32-4d44-8837-ae13ea3bea30-kube-api-access-w86sw\") pod \"test-curl-7014-8c2f98\" (UID: \"e10dcd70-fd32-4d44-8837-ae13ea3bea30\") " pod="opendatahub/test-curl-7014-8c2f98" Nov 03 02:15:20.939110 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:20.939070 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7014-8c2f98" Nov 03 02:15:21.307079 ip-10-0-136-16 kubenswrapper[5437]: W1103 02:15:21.307041 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode10dcd70_fd32_4d44_8837_ae13ea3bea30.slice/crio-ab9ae5bfcb3fa99cd244eafe488ba1e63d2493b35618f7baf362978eb99ffca5 WatchSource:0}: Error finding container ab9ae5bfcb3fa99cd244eafe488ba1e63d2493b35618f7baf362978eb99ffca5: Status 404 returned error can't find the container with id ab9ae5bfcb3fa99cd244eafe488ba1e63d2493b35618f7baf362978eb99ffca5 Nov 03 02:15:21.309759 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:21.309732 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7014-8c2f98"] Nov 03 02:15:21.473829 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:21.473789 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:15:21.500866 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:21.500833 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7014-8c2f98" event={"ID":"e10dcd70-fd32-4d44-8837-ae13ea3bea30","Type":"ContainerStarted","Data":"45082494c6baa5facf805f99a06e4b69865f2cd7fe3372e2d96c04a22f70db3a"} Nov 03 02:15:21.501360 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:21.500883 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7014-8c2f98" event={"ID":"e10dcd70-fd32-4d44-8837-ae13ea3bea30","Type":"ContainerStarted","Data":"ab9ae5bfcb3fa99cd244eafe488ba1e63d2493b35618f7baf362978eb99ffca5"} Nov 03 02:15:21.618813 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:21.618703 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7014-8c2f98" podStartSLOduration=1.618689221 podStartE2EDuration="1.618689221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:15:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:15:21.617890435 +0000 UTC m=+3403.647372730" watchObservedRunningTime="2026-11-03 02:15:21.618689221 +0000 UTC m=+3403.648171520" Nov 03 02:15:22.089240 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:22.089206 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="437aab4b-3661-46d1-a41f-8da45f894600" path="/var/lib/kubelet/pods/437aab4b-3661-46d1-a41f-8da45f894600/volumes" Nov 03 02:15:22.089693 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:22.089678 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa117337-4660-4c7b-ac91-a1a9d842bfc7" path="/var/lib/kubelet/pods/fa117337-4660-4c7b-ac91-a1a9d842bfc7/volumes" Nov 03 02:15:23.103669 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:23.103622 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7014-8c2f98"] Nov 03 02:15:23.107296 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:23.107264 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7014-8c2f98"] Nov 03 02:15:23.516290 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:23.516184 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7014-8c2f98" podUID="e10dcd70-fd32-4d44-8837-ae13ea3bea30" containerName="test-curl-7014-8c2f98" containerID="cri-o://45082494c6baa5facf805f99a06e4b69865f2cd7fe3372e2d96c04a22f70db3a" gracePeriod=2 Nov 03 02:15:23.569544 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:15:23.569505 5437 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7014-8c2f98\" is forbidden: User \"system:node:ip-10-0-136-16.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-16.us-west-2.compute.internal' and this object" podUID="e10dcd70-fd32-4d44-8837-ae13ea3bea30" pod="opendatahub/test-curl-7014-8c2f98" Nov 03 02:15:25.761654 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:25.761621 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7014-8c2f98" Nov 03 02:15:25.894238 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:25.894198 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e10dcd70-fd32-4d44-8837-ae13ea3bea30-kube-api-access-w86sw\" (UniqueName: \"kubernetes.io/projected/e10dcd70-fd32-4d44-8837-ae13ea3bea30-kube-api-access-w86sw\") pod \"e10dcd70-fd32-4d44-8837-ae13ea3bea30\" (UID: \"e10dcd70-fd32-4d44-8837-ae13ea3bea30\") " Nov 03 02:15:25.896426 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:25.896400 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e10dcd70-fd32-4d44-8837-ae13ea3bea30-kube-api-access-w86sw" pod "e10dcd70-fd32-4d44-8837-ae13ea3bea30" (UID: "e10dcd70-fd32-4d44-8837-ae13ea3bea30"). InnerVolumeSpecName "kube-api-access-w86sw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:15:25.995443 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:25.995404 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w86sw\" (UniqueName: \"kubernetes.io/projected/e10dcd70-fd32-4d44-8837-ae13ea3bea30-kube-api-access-w86sw\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:15:26.089875 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:26.089833 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e10dcd70-fd32-4d44-8837-ae13ea3bea30" path="/var/lib/kubelet/pods/e10dcd70-fd32-4d44-8837-ae13ea3bea30/volumes" Nov 03 02:15:26.210609 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:26.210525 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:15:26.546080 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:26.545987 5437 generic.go:356] "Generic (PLEG): container finished" podID="e10dcd70-fd32-4d44-8837-ae13ea3bea30" containerID="45082494c6baa5facf805f99a06e4b69865f2cd7fe3372e2d96c04a22f70db3a" exitCode=137 Nov 03 02:15:26.546080 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:26.546034 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7014-8c2f98" Nov 03 02:15:26.546310 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:26.546090 5437 scope.go:122] "RemoveContainer" containerID="45082494c6baa5facf805f99a06e4b69865f2cd7fe3372e2d96c04a22f70db3a" Nov 03 02:15:26.555443 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:26.555419 5437 scope.go:122] "RemoveContainer" containerID="45082494c6baa5facf805f99a06e4b69865f2cd7fe3372e2d96c04a22f70db3a" Nov 03 02:15:26.555791 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:15:26.555767 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45082494c6baa5facf805f99a06e4b69865f2cd7fe3372e2d96c04a22f70db3a\": container with ID starting with 45082494c6baa5facf805f99a06e4b69865f2cd7fe3372e2d96c04a22f70db3a not found: ID does not exist" containerID="45082494c6baa5facf805f99a06e4b69865f2cd7fe3372e2d96c04a22f70db3a" Nov 03 02:15:26.555854 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:26.555803 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45082494c6baa5facf805f99a06e4b69865f2cd7fe3372e2d96c04a22f70db3a"} err="failed to get container status \"45082494c6baa5facf805f99a06e4b69865f2cd7fe3372e2d96c04a22f70db3a\": rpc error: code = NotFound desc = could not find container \"45082494c6baa5facf805f99a06e4b69865f2cd7fe3372e2d96c04a22f70db3a\": container with ID starting with 45082494c6baa5facf805f99a06e4b69865f2cd7fe3372e2d96c04a22f70db3a not found: ID does not exist" Nov 03 02:15:30.954024 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:30.953981 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:15:34.386232 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:34.386193 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-b16721/explicit-model-b16721-kserve-57468d94b5-cpnnt"] Nov 03 02:15:34.386930 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:34.386907 5437 cpu_manager.go:414] "containerMap: removing container" podUID="437aab4b-3661-46d1-a41f-8da45f894600" containerName="cleanup" Nov 03 02:15:34.387026 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:34.386932 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="437aab4b-3661-46d1-a41f-8da45f894600" containerName="cleanup" Nov 03 02:15:34.387026 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:34.386971 5437 cpu_manager.go:414] "containerMap: removing container" podUID="fa117337-4660-4c7b-ac91-a1a9d842bfc7" containerName="cleanup" Nov 03 02:15:34.387026 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:34.386978 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa117337-4660-4c7b-ac91-a1a9d842bfc7" containerName="cleanup" Nov 03 02:15:34.387026 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:34.386997 5437 cpu_manager.go:414] "containerMap: removing container" podUID="e10dcd70-fd32-4d44-8837-ae13ea3bea30" containerName="test-curl-7014-8c2f98" Nov 03 02:15:34.387026 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:34.387003 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e10dcd70-fd32-4d44-8837-ae13ea3bea30" containerName="test-curl-7014-8c2f98" Nov 03 02:15:34.387312 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:34.387094 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="437aab4b-3661-46d1-a41f-8da45f894600" containerName="cleanup" Nov 03 02:15:34.387312 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:34.387104 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ad15281-5d31-4d89-bd48-185887526864" containerName="cleanup" Nov 03 02:15:34.387312 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:34.387116 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="e10dcd70-fd32-4d44-8837-ae13ea3bea30" containerName="test-curl-7014-8c2f98" Nov 03 02:15:34.392160 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:34.392137 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-b16721/explicit-model-b16721-kserve-57468d94b5-cpnnt" Nov 03 02:15:34.446504 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:34.446467 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-b16721/explicit-model-b16721-kserve-57468d94b5-cpnnt"] Nov 03 02:15:34.493835 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:34.493796 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-b16721\"/\"kube-root-ca.crt\"" Nov 03 02:15:34.494024 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:34.493956 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-b16721\"/\"default-dockercfg-t7zjb\"" Nov 03 02:15:34.494024 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:34.494005 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-b16721\"/\"openshift-service-ca.crt\"" Nov 03 02:15:34.498063 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:34.498040 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-b16721\"/\"explicit-model-b16721-kserve-self-signed-certs\"" Nov 03 02:15:34.500949 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:34.500917 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcfkd\" (UniqueName: \"kubernetes.io/projected/3170f7d2-a77b-46a6-bad8-54c051eaa8a0-kube-api-access-bcfkd\") pod \"explicit-model-b16721-kserve-57468d94b5-cpnnt\" (UID: \"3170f7d2-a77b-46a6-bad8-54c051eaa8a0\") " pod="ai-tenant-e2e-explicit-b16721/explicit-model-b16721-kserve-57468d94b5-cpnnt" Nov 03 02:15:34.501101 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:34.501000 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3170f7d2-a77b-46a6-bad8-54c051eaa8a0-tls-certs\") pod \"explicit-model-b16721-kserve-57468d94b5-cpnnt\" (UID: \"3170f7d2-a77b-46a6-bad8-54c051eaa8a0\") " pod="ai-tenant-e2e-explicit-b16721/explicit-model-b16721-kserve-57468d94b5-cpnnt" Nov 03 02:15:34.501101 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:34.501075 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3170f7d2-a77b-46a6-bad8-54c051eaa8a0-home\") pod \"explicit-model-b16721-kserve-57468d94b5-cpnnt\" (UID: \"3170f7d2-a77b-46a6-bad8-54c051eaa8a0\") " pod="ai-tenant-e2e-explicit-b16721/explicit-model-b16721-kserve-57468d94b5-cpnnt" Nov 03 02:15:34.501209 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:34.501174 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3170f7d2-a77b-46a6-bad8-54c051eaa8a0-tmp-dir\") pod \"explicit-model-b16721-kserve-57468d94b5-cpnnt\" (UID: \"3170f7d2-a77b-46a6-bad8-54c051eaa8a0\") " pod="ai-tenant-e2e-explicit-b16721/explicit-model-b16721-kserve-57468d94b5-cpnnt" Nov 03 02:15:34.501397 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:34.501349 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3170f7d2-a77b-46a6-bad8-54c051eaa8a0-model-cache\") pod \"explicit-model-b16721-kserve-57468d94b5-cpnnt\" (UID: \"3170f7d2-a77b-46a6-bad8-54c051eaa8a0\") " pod="ai-tenant-e2e-explicit-b16721/explicit-model-b16721-kserve-57468d94b5-cpnnt" Nov 03 02:15:34.501481 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:34.501450 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3170f7d2-a77b-46a6-bad8-54c051eaa8a0-dshm\") pod \"explicit-model-b16721-kserve-57468d94b5-cpnnt\" (UID: \"3170f7d2-a77b-46a6-bad8-54c051eaa8a0\") " pod="ai-tenant-e2e-explicit-b16721/explicit-model-b16721-kserve-57468d94b5-cpnnt" Nov 03 02:15:34.548027 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:34.547985 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:15:34.602609 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:34.602569 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3170f7d2-a77b-46a6-bad8-54c051eaa8a0-tmp-dir\") pod \"explicit-model-b16721-kserve-57468d94b5-cpnnt\" (UID: \"3170f7d2-a77b-46a6-bad8-54c051eaa8a0\") " pod="ai-tenant-e2e-explicit-b16721/explicit-model-b16721-kserve-57468d94b5-cpnnt" Nov 03 02:15:34.602818 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:34.602660 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3170f7d2-a77b-46a6-bad8-54c051eaa8a0-model-cache\") pod \"explicit-model-b16721-kserve-57468d94b5-cpnnt\" (UID: \"3170f7d2-a77b-46a6-bad8-54c051eaa8a0\") " pod="ai-tenant-e2e-explicit-b16721/explicit-model-b16721-kserve-57468d94b5-cpnnt" Nov 03 02:15:34.602818 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:34.602699 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3170f7d2-a77b-46a6-bad8-54c051eaa8a0-dshm\") pod \"explicit-model-b16721-kserve-57468d94b5-cpnnt\" (UID: \"3170f7d2-a77b-46a6-bad8-54c051eaa8a0\") " pod="ai-tenant-e2e-explicit-b16721/explicit-model-b16721-kserve-57468d94b5-cpnnt" Nov 03 02:15:34.602818 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:34.602775 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcfkd\" (UniqueName: \"kubernetes.io/projected/3170f7d2-a77b-46a6-bad8-54c051eaa8a0-kube-api-access-bcfkd\") pod \"explicit-model-b16721-kserve-57468d94b5-cpnnt\" (UID: \"3170f7d2-a77b-46a6-bad8-54c051eaa8a0\") " pod="ai-tenant-e2e-explicit-b16721/explicit-model-b16721-kserve-57468d94b5-cpnnt" Nov 03 02:15:34.602818 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:34.602815 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3170f7d2-a77b-46a6-bad8-54c051eaa8a0-tls-certs\") pod \"explicit-model-b16721-kserve-57468d94b5-cpnnt\" (UID: \"3170f7d2-a77b-46a6-bad8-54c051eaa8a0\") " pod="ai-tenant-e2e-explicit-b16721/explicit-model-b16721-kserve-57468d94b5-cpnnt" Nov 03 02:15:34.603044 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:34.602902 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3170f7d2-a77b-46a6-bad8-54c051eaa8a0-home\") pod \"explicit-model-b16721-kserve-57468d94b5-cpnnt\" (UID: \"3170f7d2-a77b-46a6-bad8-54c051eaa8a0\") " pod="ai-tenant-e2e-explicit-b16721/explicit-model-b16721-kserve-57468d94b5-cpnnt" Nov 03 02:15:34.603044 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:34.603023 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3170f7d2-a77b-46a6-bad8-54c051eaa8a0-tmp-dir\") pod \"explicit-model-b16721-kserve-57468d94b5-cpnnt\" (UID: \"3170f7d2-a77b-46a6-bad8-54c051eaa8a0\") " pod="ai-tenant-e2e-explicit-b16721/explicit-model-b16721-kserve-57468d94b5-cpnnt" Nov 03 02:15:34.603158 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:34.603044 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3170f7d2-a77b-46a6-bad8-54c051eaa8a0-model-cache\") pod \"explicit-model-b16721-kserve-57468d94b5-cpnnt\" (UID: \"3170f7d2-a77b-46a6-bad8-54c051eaa8a0\") " pod="ai-tenant-e2e-explicit-b16721/explicit-model-b16721-kserve-57468d94b5-cpnnt" Nov 03 02:15:34.603265 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:34.603246 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3170f7d2-a77b-46a6-bad8-54c051eaa8a0-home\") pod \"explicit-model-b16721-kserve-57468d94b5-cpnnt\" (UID: \"3170f7d2-a77b-46a6-bad8-54c051eaa8a0\") " pod="ai-tenant-e2e-explicit-b16721/explicit-model-b16721-kserve-57468d94b5-cpnnt" Nov 03 02:15:34.604976 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:34.604957 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3170f7d2-a77b-46a6-bad8-54c051eaa8a0-dshm\") pod \"explicit-model-b16721-kserve-57468d94b5-cpnnt\" (UID: \"3170f7d2-a77b-46a6-bad8-54c051eaa8a0\") " pod="ai-tenant-e2e-explicit-b16721/explicit-model-b16721-kserve-57468d94b5-cpnnt" Nov 03 02:15:34.605385 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:34.605347 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3170f7d2-a77b-46a6-bad8-54c051eaa8a0-tls-certs\") pod \"explicit-model-b16721-kserve-57468d94b5-cpnnt\" (UID: \"3170f7d2-a77b-46a6-bad8-54c051eaa8a0\") " pod="ai-tenant-e2e-explicit-b16721/explicit-model-b16721-kserve-57468d94b5-cpnnt" Nov 03 02:15:34.666228 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:34.666154 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcfkd\" (UniqueName: \"kubernetes.io/projected/3170f7d2-a77b-46a6-bad8-54c051eaa8a0-kube-api-access-bcfkd\") pod \"explicit-model-b16721-kserve-57468d94b5-cpnnt\" (UID: \"3170f7d2-a77b-46a6-bad8-54c051eaa8a0\") " pod="ai-tenant-e2e-explicit-b16721/explicit-model-b16721-kserve-57468d94b5-cpnnt" Nov 03 02:15:34.706297 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:34.706261 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-b16721/explicit-model-b16721-kserve-57468d94b5-cpnnt" Nov 03 02:15:35.082143 ip-10-0-136-16 kubenswrapper[5437]: W1103 02:15:35.082110 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3170f7d2_a77b_46a6_bad8_54c051eaa8a0.slice/crio-e16c147a66886238b6c34764d8dabe7c1fed9f64e6b4f58aabfd00c71cfa1b25 WatchSource:0}: Error finding container e16c147a66886238b6c34764d8dabe7c1fed9f64e6b4f58aabfd00c71cfa1b25: Status 404 returned error can't find the container with id e16c147a66886238b6c34764d8dabe7c1fed9f64e6b4f58aabfd00c71cfa1b25 Nov 03 02:15:35.085193 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:35.085172 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-b16721/explicit-model-b16721-kserve-57468d94b5-cpnnt"] Nov 03 02:15:35.608601 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:35.608561 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-b16721/explicit-model-b16721-kserve-57468d94b5-cpnnt" event={"ID":"3170f7d2-a77b-46a6-bad8-54c051eaa8a0","Type":"ContainerStarted","Data":"3e5b10dbf1bb951411f5604bb25aa77e06a70cf47dab5c59ac27ab5e74d26a39"} Nov 03 02:15:35.609178 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:35.609141 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-b16721/explicit-model-b16721-kserve-57468d94b5-cpnnt" event={"ID":"3170f7d2-a77b-46a6-bad8-54c051eaa8a0","Type":"ContainerStarted","Data":"e16c147a66886238b6c34764d8dabe7c1fed9f64e6b4f58aabfd00c71cfa1b25"} Nov 03 02:15:35.706905 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:35.706870 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-b16721/explicit-model-b16721-kserve-57468d94b5-cpnnt" Nov 03 02:15:35.733255 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:35.733199 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-b16721/explicit-model-b16721-kserve-57468d94b5-cpnnt" podStartSLOduration=1.733180632 podStartE2EDuration="1.733180632s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:15:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:15:35.731760632 +0000 UTC m=+3417.761242926" watchObservedRunningTime="2026-11-03 02:15:35.733180632 +0000 UTC m=+3417.762662926" Nov 03 02:15:35.765724 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:35.765689 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:15:40.519213 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:40.519178 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:15:44.035703 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:15:44.035658 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 02:15:44.707119 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:44.707083 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-b16721/explicit-model-b16721-kserve-57468d94b5-cpnnt" Nov 03 02:15:44.721348 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:44.721318 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-b16721/explicit-model-b16721-kserve-57468d94b5-cpnnt" Nov 03 02:15:45.271005 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:45.270968 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:15:45.692492 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:45.692463 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-b16721/explicit-model-b16721-kserve-57468d94b5-cpnnt" Nov 03 02:15:50.006137 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:50.006045 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:15:52.990705 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:52.990670 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-a54cab/auth-test-model-a54cab-kserve-7db9d647d9-pddx2"] Nov 03 02:15:53.056836 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:53.056793 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:15:54.790079 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:54.790039 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:15:55.299130 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:55.299093 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:15:55.496109 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:55.496071 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:15:55.978634 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:55.978602 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-b16721/explicit-model-b16721-kserve-57468d94b5-cpnnt"] Nov 03 02:15:56.049459 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:56.049409 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:15:56.492209 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:56.492158 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:15:57.006112 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:57.006073 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:15:57.170956 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:57.170923 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7005-b341a2"] Nov 03 02:15:57.176841 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:57.176815 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7005-b341a2" Nov 03 02:15:57.226061 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:57.226016 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7005-b341a2"] Nov 03 02:15:57.273690 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:57.273607 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-882bv\"" Nov 03 02:15:57.284271 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:57.284232 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6822l\" (UniqueName: \"kubernetes.io/projected/8574401d-2b12-4973-bf48-84fd5d166e84-kube-api-access-6822l\") pod \"test-curl-7005-b341a2\" (UID: \"8574401d-2b12-4973-bf48-84fd5d166e84\") " pod="opendatahub/test-curl-7005-b341a2" Nov 03 02:15:57.385956 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:57.385920 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6822l\" (UniqueName: \"kubernetes.io/projected/8574401d-2b12-4973-bf48-84fd5d166e84-kube-api-access-6822l\") pod \"test-curl-7005-b341a2\" (UID: \"8574401d-2b12-4973-bf48-84fd5d166e84\") " pod="opendatahub/test-curl-7005-b341a2" Nov 03 02:15:57.448908 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:57.448876 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6822l\" (UniqueName: \"kubernetes.io/projected/8574401d-2b12-4973-bf48-84fd5d166e84-kube-api-access-6822l\") pod \"test-curl-7005-b341a2\" (UID: \"8574401d-2b12-4973-bf48-84fd5d166e84\") " pod="opendatahub/test-curl-7005-b341a2" Nov 03 02:15:57.490205 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:57.490167 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7005-b341a2" Nov 03 02:15:57.865611 ip-10-0-136-16 kubenswrapper[5437]: W1103 02:15:57.865576 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8574401d_2b12_4973_bf48_84fd5d166e84.slice/crio-892b76133fd081c0f49f59b416ba8a60ca9a4c4126893e9f789baf41eabcfe26 WatchSource:0}: Error finding container 892b76133fd081c0f49f59b416ba8a60ca9a4c4126893e9f789baf41eabcfe26: Status 404 returned error can't find the container with id 892b76133fd081c0f49f59b416ba8a60ca9a4c4126893e9f789baf41eabcfe26 Nov 03 02:15:57.867665 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:57.867638 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7005-b341a2"] Nov 03 02:15:57.956081 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:57.956042 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:15:58.766119 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:58.766083 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7005-b341a2" event={"ID":"8574401d-2b12-4973-bf48-84fd5d166e84","Type":"ContainerStarted","Data":"a6268c20c554c104db195fc7383f3af785b072e714763054f7a8b35c61255ee3"} Nov 03 02:15:58.768426 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:58.766128 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7005-b341a2" event={"ID":"8574401d-2b12-4973-bf48-84fd5d166e84","Type":"ContainerStarted","Data":"892b76133fd081c0f49f59b416ba8a60ca9a4c4126893e9f789baf41eabcfe26"} Nov 03 02:15:58.884149 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:58.884098 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7005-b341a2" podStartSLOduration=1.884085456 podStartE2EDuration="1.884085456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:15:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:15:58.882297121 +0000 UTC m=+3440.911779417" watchObservedRunningTime="2026-11-03 02:15:58.884085456 +0000 UTC m=+3440.913567751" Nov 03 02:15:58.962428 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:58.962357 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:15:59.546434 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:59.546401 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:15:59.752438 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:15:59.752401 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:16:00.110542 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:00.110508 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:16:00.321012 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:00.320977 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7005-b341a2"] Nov 03 02:16:00.324408 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:00.324350 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7005-b341a2"] Nov 03 02:16:00.781667 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:00.781617 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7005-b341a2" podUID="8574401d-2b12-4973-bf48-84fd5d166e84" containerName="test-curl-7005-b341a2" containerID="cri-o://a6268c20c554c104db195fc7383f3af785b072e714763054f7a8b35c61255ee3" gracePeriod=2 Nov 03 02:16:00.834827 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:16:00.834780 5437 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7005-b341a2\" is forbidden: User \"system:node:ip-10-0-136-16.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-16.us-west-2.compute.internal' and this object" podUID="8574401d-2b12-4973-bf48-84fd5d166e84" pod="opendatahub/test-curl-7005-b341a2" Nov 03 02:16:01.695230 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:01.695194 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:16:01.951694 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:01.951597 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:16:02.581423 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:02.581340 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:16:02.934297 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:02.934263 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:16:03.046870 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:03.046841 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7005-b341a2" Nov 03 02:16:03.150672 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:03.150582 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8574401d-2b12-4973-bf48-84fd5d166e84-kube-api-access-6822l\" (UniqueName: \"kubernetes.io/projected/8574401d-2b12-4973-bf48-84fd5d166e84-kube-api-access-6822l\") pod \"8574401d-2b12-4973-bf48-84fd5d166e84\" (UID: \"8574401d-2b12-4973-bf48-84fd5d166e84\") " Nov 03 02:16:03.152849 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:03.152820 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8574401d-2b12-4973-bf48-84fd5d166e84-kube-api-access-6822l" pod "8574401d-2b12-4973-bf48-84fd5d166e84" (UID: "8574401d-2b12-4973-bf48-84fd5d166e84"). InnerVolumeSpecName "kube-api-access-6822l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:16:03.251740 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:03.251711 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6822l\" (UniqueName: \"kubernetes.io/projected/8574401d-2b12-4973-bf48-84fd5d166e84-kube-api-access-6822l\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:16:03.802333 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:03.802293 5437 generic.go:356] "Generic (PLEG): container finished" podID="8574401d-2b12-4973-bf48-84fd5d166e84" containerID="a6268c20c554c104db195fc7383f3af785b072e714763054f7a8b35c61255ee3" exitCode=137 Nov 03 02:16:03.802554 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:03.802342 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7005-b341a2" Nov 03 02:16:03.802554 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:03.802407 5437 scope.go:122] "RemoveContainer" containerID="a6268c20c554c104db195fc7383f3af785b072e714763054f7a8b35c61255ee3" Nov 03 02:16:03.812020 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:03.811995 5437 scope.go:122] "RemoveContainer" containerID="a6268c20c554c104db195fc7383f3af785b072e714763054f7a8b35c61255ee3" Nov 03 02:16:03.812338 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:16:03.812317 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6268c20c554c104db195fc7383f3af785b072e714763054f7a8b35c61255ee3\": container with ID starting with a6268c20c554c104db195fc7383f3af785b072e714763054f7a8b35c61255ee3 not found: ID does not exist" containerID="a6268c20c554c104db195fc7383f3af785b072e714763054f7a8b35c61255ee3" Nov 03 02:16:03.812482 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:03.812348 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6268c20c554c104db195fc7383f3af785b072e714763054f7a8b35c61255ee3"} err="failed to get container status \"a6268c20c554c104db195fc7383f3af785b072e714763054f7a8b35c61255ee3\": rpc error: code = NotFound desc = could not find container \"a6268c20c554c104db195fc7383f3af785b072e714763054f7a8b35c61255ee3\": container with ID starting with a6268c20c554c104db195fc7383f3af785b072e714763054f7a8b35c61255ee3 not found: ID does not exist" Nov 03 02:16:03.863867 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:16:03.863823 5437 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7005-b341a2\" is forbidden: User \"system:node:ip-10-0-136-16.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-16.us-west-2.compute.internal' and this object" podUID="8574401d-2b12-4973-bf48-84fd5d166e84" pod="opendatahub/test-curl-7005-b341a2" Nov 03 02:16:04.089717 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:04.089638 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8574401d-2b12-4973-bf48-84fd5d166e84" path="/var/lib/kubelet/pods/8574401d-2b12-4973-bf48-84fd5d166e84/volumes" Nov 03 02:16:04.576947 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:04.576905 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:16:06.690184 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:06.690138 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:16:06.899839 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:06.899803 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:16:08.013895 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:08.013856 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w1-a54cab/auth-test-model-a54cab-kserve-7db9d647d9-pddx2" podUID="6ee8734a-4bb3-4efd-b601-797918ce5a9f" containerName="main" containerID="cri-o://58de8d65e8cf0e8868f3ad3411065da6c01318053dd908475369a70535c00dfe" gracePeriod=45 Nov 03 02:16:08.039716 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:08.039681 5437 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w1-a54cab/auth-test-model-a54cab-kserve-7db9d647d9-pddx2" podUID="6ee8734a-4bb3-4efd-b601-797918ce5a9f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.142:8000/ready\": dial tcp 10.134.0.142:8000: connect: connection refused" Nov 03 02:16:08.269263 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:08.269198 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-a54cab/auth-test-model-a54cab-kserve-7db9d647d9-pddx2" Nov 03 02:16:08.313915 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:08.313877 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ee8734a-4bb3-4efd-b601-797918ce5a9f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ee8734a-4bb3-4efd-b601-797918ce5a9f-tmp-dir\") pod \"6ee8734a-4bb3-4efd-b601-797918ce5a9f\" (UID: \"6ee8734a-4bb3-4efd-b601-797918ce5a9f\") " Nov 03 02:16:08.314086 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:08.313988 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ee8734a-4bb3-4efd-b601-797918ce5a9f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ee8734a-4bb3-4efd-b601-797918ce5a9f-dshm\") pod \"6ee8734a-4bb3-4efd-b601-797918ce5a9f\" (UID: \"6ee8734a-4bb3-4efd-b601-797918ce5a9f\") " Nov 03 02:16:08.314086 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:08.314019 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ee8734a-4bb3-4efd-b601-797918ce5a9f-tls-certs\" (UniqueName: \"kubernetes.io/secret/6ee8734a-4bb3-4efd-b601-797918ce5a9f-tls-certs\") pod \"6ee8734a-4bb3-4efd-b601-797918ce5a9f\" (UID: \"6ee8734a-4bb3-4efd-b601-797918ce5a9f\") " Nov 03 02:16:08.314165 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:08.314115 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ee8734a-4bb3-4efd-b601-797918ce5a9f-kube-api-access-f7k62\" (UniqueName: \"kubernetes.io/projected/6ee8734a-4bb3-4efd-b601-797918ce5a9f-kube-api-access-f7k62\") pod \"6ee8734a-4bb3-4efd-b601-797918ce5a9f\" (UID: \"6ee8734a-4bb3-4efd-b601-797918ce5a9f\") " Nov 03 02:16:08.314165 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:08.314155 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ee8734a-4bb3-4efd-b601-797918ce5a9f-home\" (UniqueName: \"kubernetes.io/empty-dir/6ee8734a-4bb3-4efd-b601-797918ce5a9f-home\") pod \"6ee8734a-4bb3-4efd-b601-797918ce5a9f\" (UID: \"6ee8734a-4bb3-4efd-b601-797918ce5a9f\") " Nov 03 02:16:08.314235 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:08.314193 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ee8734a-4bb3-4efd-b601-797918ce5a9f-tmp-dir" pod "6ee8734a-4bb3-4efd-b601-797918ce5a9f" (UID: "6ee8734a-4bb3-4efd-b601-797918ce5a9f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:16:08.314288 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:08.314249 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ee8734a-4bb3-4efd-b601-797918ce5a9f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ee8734a-4bb3-4efd-b601-797918ce5a9f-model-cache\") pod \"6ee8734a-4bb3-4efd-b601-797918ce5a9f\" (UID: \"6ee8734a-4bb3-4efd-b601-797918ce5a9f\") " Nov 03 02:16:08.314536 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:08.314508 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ee8734a-4bb3-4efd-b601-797918ce5a9f-home" pod "6ee8734a-4bb3-4efd-b601-797918ce5a9f" (UID: "6ee8734a-4bb3-4efd-b601-797918ce5a9f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:16:08.314667 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:08.314549 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ee8734a-4bb3-4efd-b601-797918ce5a9f-model-cache" pod "6ee8734a-4bb3-4efd-b601-797918ce5a9f" (UID: "6ee8734a-4bb3-4efd-b601-797918ce5a9f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:16:08.314862 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:08.314846 5437 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ee8734a-4bb3-4efd-b601-797918ce5a9f-home\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:16:08.314907 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:08.314867 5437 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ee8734a-4bb3-4efd-b601-797918ce5a9f-model-cache\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:16:08.314907 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:08.314876 5437 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ee8734a-4bb3-4efd-b601-797918ce5a9f-tmp-dir\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:16:08.316442 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:08.316418 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ee8734a-4bb3-4efd-b601-797918ce5a9f-tls-certs" pod "6ee8734a-4bb3-4efd-b601-797918ce5a9f" (UID: "6ee8734a-4bb3-4efd-b601-797918ce5a9f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:16:08.316624 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:08.316603 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ee8734a-4bb3-4efd-b601-797918ce5a9f-dshm" pod "6ee8734a-4bb3-4efd-b601-797918ce5a9f" (UID: "6ee8734a-4bb3-4efd-b601-797918ce5a9f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:16:08.316699 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:08.316630 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ee8734a-4bb3-4efd-b601-797918ce5a9f-kube-api-access-f7k62" pod "6ee8734a-4bb3-4efd-b601-797918ce5a9f" (UID: "6ee8734a-4bb3-4efd-b601-797918ce5a9f"). InnerVolumeSpecName "kube-api-access-f7k62". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:16:08.415828 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:08.415790 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7k62\" (UniqueName: \"kubernetes.io/projected/6ee8734a-4bb3-4efd-b601-797918ce5a9f-kube-api-access-f7k62\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:16:08.415828 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:08.415822 5437 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ee8734a-4bb3-4efd-b601-797918ce5a9f-dshm\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:16:08.415828 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:08.415834 5437 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ee8734a-4bb3-4efd-b601-797918ce5a9f-tls-certs\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:16:08.835110 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:08.835075 5437 generic.go:356] "Generic (PLEG): container finished" podID="6ee8734a-4bb3-4efd-b601-797918ce5a9f" containerID="58de8d65e8cf0e8868f3ad3411065da6c01318053dd908475369a70535c00dfe" exitCode=0 Nov 03 02:16:08.835298 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:08.835126 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-a54cab/auth-test-model-a54cab-kserve-7db9d647d9-pddx2" Nov 03 02:16:08.835298 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:08.835163 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-a54cab/auth-test-model-a54cab-kserve-7db9d647d9-pddx2" event={"ID":"6ee8734a-4bb3-4efd-b601-797918ce5a9f","Type":"ContainerDied","Data":"58de8d65e8cf0e8868f3ad3411065da6c01318053dd908475369a70535c00dfe"} Nov 03 02:16:08.835298 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:08.835209 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-a54cab/auth-test-model-a54cab-kserve-7db9d647d9-pddx2" event={"ID":"6ee8734a-4bb3-4efd-b601-797918ce5a9f","Type":"ContainerDied","Data":"6110c32ad0da13328481618dbffe77106c283e03f4583cd5d9aab4f6c15c900b"} Nov 03 02:16:08.835298 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:08.835233 5437 scope.go:122] "RemoveContainer" containerID="58de8d65e8cf0e8868f3ad3411065da6c01318053dd908475369a70535c00dfe" Nov 03 02:16:08.848719 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:08.848697 5437 scope.go:122] "RemoveContainer" containerID="58de8d65e8cf0e8868f3ad3411065da6c01318053dd908475369a70535c00dfe" Nov 03 02:16:08.848992 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:16:08.848969 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58de8d65e8cf0e8868f3ad3411065da6c01318053dd908475369a70535c00dfe\": container with ID starting with 58de8d65e8cf0e8868f3ad3411065da6c01318053dd908475369a70535c00dfe not found: ID does not exist" containerID="58de8d65e8cf0e8868f3ad3411065da6c01318053dd908475369a70535c00dfe" Nov 03 02:16:08.849059 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:08.849005 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58de8d65e8cf0e8868f3ad3411065da6c01318053dd908475369a70535c00dfe"} err="failed to get container status \"58de8d65e8cf0e8868f3ad3411065da6c01318053dd908475369a70535c00dfe\": rpc error: code = NotFound desc = could not find container \"58de8d65e8cf0e8868f3ad3411065da6c01318053dd908475369a70535c00dfe\": container with ID starting with 58de8d65e8cf0e8868f3ad3411065da6c01318053dd908475369a70535c00dfe not found: ID does not exist" Nov 03 02:16:09.020219 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:09.020186 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-a54cab/auth-test-model-a54cab-kserve-7db9d647d9-pddx2"] Nov 03 02:16:09.025958 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:09.025933 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w1-a54cab/auth-test-model-a54cab-kserve-7db9d647d9-pddx2"] Nov 03 02:16:09.668794 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:09.668758 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:16:10.090358 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:10.090320 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ee8734a-4bb3-4efd-b601-797918ce5a9f" path="/var/lib/kubelet/pods/6ee8734a-4bb3-4efd-b601-797918ce5a9f/volumes" Nov 03 02:16:10.999474 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:10.999430 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-b16721/explicit-model-b16721-kserve-57468d94b5-cpnnt" podUID="3170f7d2-a77b-46a6-bad8-54c051eaa8a0" containerName="main" containerID="cri-o://3e5b10dbf1bb951411f5604bb25aa77e06a70cf47dab5c59ac27ab5e74d26a39" gracePeriod=45 Nov 03 02:16:11.271435 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:11.271409 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-b16721/explicit-model-b16721-kserve-57468d94b5-cpnnt" Nov 03 02:16:11.353874 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:11.353842 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3170f7d2-a77b-46a6-bad8-54c051eaa8a0-kube-api-access-bcfkd\" (UniqueName: \"kubernetes.io/projected/3170f7d2-a77b-46a6-bad8-54c051eaa8a0-kube-api-access-bcfkd\") pod \"3170f7d2-a77b-46a6-bad8-54c051eaa8a0\" (UID: \"3170f7d2-a77b-46a6-bad8-54c051eaa8a0\") " Nov 03 02:16:11.354061 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:11.353918 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3170f7d2-a77b-46a6-bad8-54c051eaa8a0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3170f7d2-a77b-46a6-bad8-54c051eaa8a0-model-cache\") pod \"3170f7d2-a77b-46a6-bad8-54c051eaa8a0\" (UID: \"3170f7d2-a77b-46a6-bad8-54c051eaa8a0\") " Nov 03 02:16:11.354061 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:11.353971 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3170f7d2-a77b-46a6-bad8-54c051eaa8a0-tls-certs\" (UniqueName: \"kubernetes.io/secret/3170f7d2-a77b-46a6-bad8-54c051eaa8a0-tls-certs\") pod \"3170f7d2-a77b-46a6-bad8-54c051eaa8a0\" (UID: \"3170f7d2-a77b-46a6-bad8-54c051eaa8a0\") " Nov 03 02:16:11.354061 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:11.353998 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3170f7d2-a77b-46a6-bad8-54c051eaa8a0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3170f7d2-a77b-46a6-bad8-54c051eaa8a0-tmp-dir\") pod \"3170f7d2-a77b-46a6-bad8-54c051eaa8a0\" (UID: \"3170f7d2-a77b-46a6-bad8-54c051eaa8a0\") " Nov 03 02:16:11.354249 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:11.354090 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3170f7d2-a77b-46a6-bad8-54c051eaa8a0-home\" (UniqueName: \"kubernetes.io/empty-dir/3170f7d2-a77b-46a6-bad8-54c051eaa8a0-home\") pod \"3170f7d2-a77b-46a6-bad8-54c051eaa8a0\" (UID: \"3170f7d2-a77b-46a6-bad8-54c051eaa8a0\") " Nov 03 02:16:11.354249 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:11.354160 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3170f7d2-a77b-46a6-bad8-54c051eaa8a0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3170f7d2-a77b-46a6-bad8-54c051eaa8a0-dshm\") pod \"3170f7d2-a77b-46a6-bad8-54c051eaa8a0\" (UID: \"3170f7d2-a77b-46a6-bad8-54c051eaa8a0\") " Nov 03 02:16:11.354249 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:11.354219 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3170f7d2-a77b-46a6-bad8-54c051eaa8a0-model-cache" pod "3170f7d2-a77b-46a6-bad8-54c051eaa8a0" (UID: "3170f7d2-a77b-46a6-bad8-54c051eaa8a0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:16:11.354432 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:11.354375 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3170f7d2-a77b-46a6-bad8-54c051eaa8a0-tmp-dir" pod "3170f7d2-a77b-46a6-bad8-54c051eaa8a0" (UID: "3170f7d2-a77b-46a6-bad8-54c051eaa8a0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:16:11.354501 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:11.354476 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3170f7d2-a77b-46a6-bad8-54c051eaa8a0-home" pod "3170f7d2-a77b-46a6-bad8-54c051eaa8a0" (UID: "3170f7d2-a77b-46a6-bad8-54c051eaa8a0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:16:11.354726 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:11.354699 5437 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3170f7d2-a77b-46a6-bad8-54c051eaa8a0-model-cache\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:16:11.354837 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:11.354729 5437 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3170f7d2-a77b-46a6-bad8-54c051eaa8a0-tmp-dir\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:16:11.354837 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:11.354745 5437 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3170f7d2-a77b-46a6-bad8-54c051eaa8a0-home\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:16:11.356434 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:11.356412 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3170f7d2-a77b-46a6-bad8-54c051eaa8a0-dshm" pod "3170f7d2-a77b-46a6-bad8-54c051eaa8a0" (UID: "3170f7d2-a77b-46a6-bad8-54c051eaa8a0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:16:11.356581 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:11.356566 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3170f7d2-a77b-46a6-bad8-54c051eaa8a0-kube-api-access-bcfkd" pod "3170f7d2-a77b-46a6-bad8-54c051eaa8a0" (UID: "3170f7d2-a77b-46a6-bad8-54c051eaa8a0"). InnerVolumeSpecName "kube-api-access-bcfkd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:16:11.356642 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:11.356619 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3170f7d2-a77b-46a6-bad8-54c051eaa8a0-tls-certs" pod "3170f7d2-a77b-46a6-bad8-54c051eaa8a0" (UID: "3170f7d2-a77b-46a6-bad8-54c051eaa8a0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:16:11.455492 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:11.455452 5437 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3170f7d2-a77b-46a6-bad8-54c051eaa8a0-dshm\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:16:11.455492 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:11.455482 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bcfkd\" (UniqueName: \"kubernetes.io/projected/3170f7d2-a77b-46a6-bad8-54c051eaa8a0-kube-api-access-bcfkd\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:16:11.455492 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:11.455492 5437 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3170f7d2-a77b-46a6-bad8-54c051eaa8a0-tls-certs\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:16:11.858453 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:11.858414 5437 generic.go:356] "Generic (PLEG): container finished" podID="3170f7d2-a77b-46a6-bad8-54c051eaa8a0" containerID="3e5b10dbf1bb951411f5604bb25aa77e06a70cf47dab5c59ac27ab5e74d26a39" exitCode=0 Nov 03 02:16:11.858792 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:11.858461 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-b16721/explicit-model-b16721-kserve-57468d94b5-cpnnt" Nov 03 02:16:11.858942 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:11.858461 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-b16721/explicit-model-b16721-kserve-57468d94b5-cpnnt" event={"ID":"3170f7d2-a77b-46a6-bad8-54c051eaa8a0","Type":"ContainerDied","Data":"3e5b10dbf1bb951411f5604bb25aa77e06a70cf47dab5c59ac27ab5e74d26a39"} Nov 03 02:16:11.859008 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:11.858947 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-b16721/explicit-model-b16721-kserve-57468d94b5-cpnnt" event={"ID":"3170f7d2-a77b-46a6-bad8-54c051eaa8a0","Type":"ContainerDied","Data":"e16c147a66886238b6c34764d8dabe7c1fed9f64e6b4f58aabfd00c71cfa1b25"} Nov 03 02:16:11.859008 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:11.858976 5437 scope.go:122] "RemoveContainer" containerID="3e5b10dbf1bb951411f5604bb25aa77e06a70cf47dab5c59ac27ab5e74d26a39" Nov 03 02:16:11.871031 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:11.871010 5437 scope.go:122] "RemoveContainer" containerID="3e5b10dbf1bb951411f5604bb25aa77e06a70cf47dab5c59ac27ab5e74d26a39" Nov 03 02:16:11.871399 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:16:11.871377 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e5b10dbf1bb951411f5604bb25aa77e06a70cf47dab5c59ac27ab5e74d26a39\": container with ID starting with 3e5b10dbf1bb951411f5604bb25aa77e06a70cf47dab5c59ac27ab5e74d26a39 not found: ID does not exist" containerID="3e5b10dbf1bb951411f5604bb25aa77e06a70cf47dab5c59ac27ab5e74d26a39" Nov 03 02:16:11.871459 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:11.871410 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e5b10dbf1bb951411f5604bb25aa77e06a70cf47dab5c59ac27ab5e74d26a39"} err="failed to get container status \"3e5b10dbf1bb951411f5604bb25aa77e06a70cf47dab5c59ac27ab5e74d26a39\": rpc error: code = NotFound desc = could not find container \"3e5b10dbf1bb951411f5604bb25aa77e06a70cf47dab5c59ac27ab5e74d26a39\": container with ID starting with 3e5b10dbf1bb951411f5604bb25aa77e06a70cf47dab5c59ac27ab5e74d26a39 not found: ID does not exist" Nov 03 02:16:12.028825 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:12.028789 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-b16721/explicit-model-b16721-kserve-57468d94b5-cpnnt"] Nov 03 02:16:12.080637 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:12.080602 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-b16721/explicit-model-b16721-kserve-57468d94b5-cpnnt"] Nov 03 02:16:12.089115 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:12.089082 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3170f7d2-a77b-46a6-bad8-54c051eaa8a0" path="/var/lib/kubelet/pods/3170f7d2-a77b-46a6-bad8-54c051eaa8a0/volumes" Nov 03 02:16:12.938847 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:12.938812 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-c34879-openshift-default-ccddfffff-f9pm9"] Nov 03 02:16:12.939437 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:12.939416 5437 cpu_manager.go:414] "containerMap: removing container" podUID="8574401d-2b12-4973-bf48-84fd5d166e84" containerName="test-curl-7005-b341a2" Nov 03 02:16:12.939437 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:12.939436 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8574401d-2b12-4973-bf48-84fd5d166e84" containerName="test-curl-7005-b341a2" Nov 03 02:16:12.939598 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:12.939459 5437 cpu_manager.go:414] "containerMap: removing container" podUID="3170f7d2-a77b-46a6-bad8-54c051eaa8a0" containerName="main" Nov 03 02:16:12.939598 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:12.939466 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3170f7d2-a77b-46a6-bad8-54c051eaa8a0" containerName="main" Nov 03 02:16:12.939598 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:12.939492 5437 cpu_manager.go:414] "containerMap: removing container" podUID="6ee8734a-4bb3-4efd-b601-797918ce5a9f" containerName="main" Nov 03 02:16:12.939598 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:12.939507 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ee8734a-4bb3-4efd-b601-797918ce5a9f" containerName="main" Nov 03 02:16:12.939801 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:12.939621 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="8574401d-2b12-4973-bf48-84fd5d166e84" containerName="test-curl-7005-b341a2" Nov 03 02:16:12.939801 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:12.939634 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="3170f7d2-a77b-46a6-bad8-54c051eaa8a0" containerName="main" Nov 03 02:16:12.939801 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:12.939644 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ee8734a-4bb3-4efd-b601-797918ce5a9f" containerName="main" Nov 03 02:16:12.944393 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:12.944353 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-c34879-openshift-default-ccddfffff-f9pm9" Nov 03 02:16:12.998962 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:12.998928 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-c34879-openshift-default-dockercfg-m95fd\"" Nov 03 02:16:13.062952 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:13.062912 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-c34879-openshift-default-ccddfffff-f9pm9"] Nov 03 02:16:13.072329 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:13.072296 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6c263e1-c60a-437a-a13a-579a420a833a-istio-podinfo\") pod \"e2e-shared-a-w4-c34879-openshift-default-ccddfffff-f9pm9\" (UID: \"f6c263e1-c60a-437a-a13a-579a420a833a\") " pod="openshift-ingress/e2e-shared-a-w4-c34879-openshift-default-ccddfffff-f9pm9" Nov 03 02:16:13.072565 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:13.072440 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6c263e1-c60a-437a-a13a-579a420a833a-istio-envoy\") pod \"e2e-shared-a-w4-c34879-openshift-default-ccddfffff-f9pm9\" (UID: \"f6c263e1-c60a-437a-a13a-579a420a833a\") " pod="openshift-ingress/e2e-shared-a-w4-c34879-openshift-default-ccddfffff-f9pm9" Nov 03 02:16:13.072565 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:13.072495 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6c263e1-c60a-437a-a13a-579a420a833a-credential-socket\") pod \"e2e-shared-a-w4-c34879-openshift-default-ccddfffff-f9pm9\" (UID: \"f6c263e1-c60a-437a-a13a-579a420a833a\") " pod="openshift-ingress/e2e-shared-a-w4-c34879-openshift-default-ccddfffff-f9pm9" Nov 03 02:16:13.072565 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:13.072530 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6c263e1-c60a-437a-a13a-579a420a833a-workload-certs\") pod \"e2e-shared-a-w4-c34879-openshift-default-ccddfffff-f9pm9\" (UID: \"f6c263e1-c60a-437a-a13a-579a420a833a\") " pod="openshift-ingress/e2e-shared-a-w4-c34879-openshift-default-ccddfffff-f9pm9" Nov 03 02:16:13.072689 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:13.072634 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rlgf\" (UniqueName: \"kubernetes.io/projected/f6c263e1-c60a-437a-a13a-579a420a833a-kube-api-access-4rlgf\") pod \"e2e-shared-a-w4-c34879-openshift-default-ccddfffff-f9pm9\" (UID: \"f6c263e1-c60a-437a-a13a-579a420a833a\") " pod="openshift-ingress/e2e-shared-a-w4-c34879-openshift-default-ccddfffff-f9pm9" Nov 03 02:16:13.072725 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:13.072701 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6c263e1-c60a-437a-a13a-579a420a833a-istiod-ca-cert\") pod \"e2e-shared-a-w4-c34879-openshift-default-ccddfffff-f9pm9\" (UID: \"f6c263e1-c60a-437a-a13a-579a420a833a\") " pod="openshift-ingress/e2e-shared-a-w4-c34879-openshift-default-ccddfffff-f9pm9" Nov 03 02:16:13.072786 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:13.072771 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6c263e1-c60a-437a-a13a-579a420a833a-workload-socket\") pod \"e2e-shared-a-w4-c34879-openshift-default-ccddfffff-f9pm9\" (UID: \"f6c263e1-c60a-437a-a13a-579a420a833a\") " pod="openshift-ingress/e2e-shared-a-w4-c34879-openshift-default-ccddfffff-f9pm9" Nov 03 02:16:13.072822 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:13.072812 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6c263e1-c60a-437a-a13a-579a420a833a-istio-token\") pod \"e2e-shared-a-w4-c34879-openshift-default-ccddfffff-f9pm9\" (UID: \"f6c263e1-c60a-437a-a13a-579a420a833a\") " pod="openshift-ingress/e2e-shared-a-w4-c34879-openshift-default-ccddfffff-f9pm9" Nov 03 02:16:13.072873 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:13.072861 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6c263e1-c60a-437a-a13a-579a420a833a-istio-data\") pod \"e2e-shared-a-w4-c34879-openshift-default-ccddfffff-f9pm9\" (UID: \"f6c263e1-c60a-437a-a13a-579a420a833a\") " pod="openshift-ingress/e2e-shared-a-w4-c34879-openshift-default-ccddfffff-f9pm9" Nov 03 02:16:13.173858 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:13.173822 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6c263e1-c60a-437a-a13a-579a420a833a-istio-podinfo\") pod \"e2e-shared-a-w4-c34879-openshift-default-ccddfffff-f9pm9\" (UID: \"f6c263e1-c60a-437a-a13a-579a420a833a\") " pod="openshift-ingress/e2e-shared-a-w4-c34879-openshift-default-ccddfffff-f9pm9" Nov 03 02:16:13.174037 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:13.173978 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6c263e1-c60a-437a-a13a-579a420a833a-istio-envoy\") pod \"e2e-shared-a-w4-c34879-openshift-default-ccddfffff-f9pm9\" (UID: \"f6c263e1-c60a-437a-a13a-579a420a833a\") " pod="openshift-ingress/e2e-shared-a-w4-c34879-openshift-default-ccddfffff-f9pm9" Nov 03 02:16:13.174037 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:13.174021 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6c263e1-c60a-437a-a13a-579a420a833a-credential-socket\") pod \"e2e-shared-a-w4-c34879-openshift-default-ccddfffff-f9pm9\" (UID: \"f6c263e1-c60a-437a-a13a-579a420a833a\") " pod="openshift-ingress/e2e-shared-a-w4-c34879-openshift-default-ccddfffff-f9pm9" Nov 03 02:16:13.174171 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:13.174056 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6c263e1-c60a-437a-a13a-579a420a833a-workload-certs\") pod \"e2e-shared-a-w4-c34879-openshift-default-ccddfffff-f9pm9\" (UID: \"f6c263e1-c60a-437a-a13a-579a420a833a\") " pod="openshift-ingress/e2e-shared-a-w4-c34879-openshift-default-ccddfffff-f9pm9" Nov 03 02:16:13.174171 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:13.174091 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rlgf\" (UniqueName: \"kubernetes.io/projected/f6c263e1-c60a-437a-a13a-579a420a833a-kube-api-access-4rlgf\") pod \"e2e-shared-a-w4-c34879-openshift-default-ccddfffff-f9pm9\" (UID: \"f6c263e1-c60a-437a-a13a-579a420a833a\") " pod="openshift-ingress/e2e-shared-a-w4-c34879-openshift-default-ccddfffff-f9pm9" Nov 03 02:16:13.174171 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:13.174155 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6c263e1-c60a-437a-a13a-579a420a833a-istiod-ca-cert\") pod \"e2e-shared-a-w4-c34879-openshift-default-ccddfffff-f9pm9\" (UID: \"f6c263e1-c60a-437a-a13a-579a420a833a\") " pod="openshift-ingress/e2e-shared-a-w4-c34879-openshift-default-ccddfffff-f9pm9" Nov 03 02:16:13.174330 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:13.174219 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6c263e1-c60a-437a-a13a-579a420a833a-workload-socket\") pod \"e2e-shared-a-w4-c34879-openshift-default-ccddfffff-f9pm9\" (UID: \"f6c263e1-c60a-437a-a13a-579a420a833a\") " pod="openshift-ingress/e2e-shared-a-w4-c34879-openshift-default-ccddfffff-f9pm9" Nov 03 02:16:13.174330 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:13.174257 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6c263e1-c60a-437a-a13a-579a420a833a-istio-token\") pod \"e2e-shared-a-w4-c34879-openshift-default-ccddfffff-f9pm9\" (UID: \"f6c263e1-c60a-437a-a13a-579a420a833a\") " pod="openshift-ingress/e2e-shared-a-w4-c34879-openshift-default-ccddfffff-f9pm9" Nov 03 02:16:13.174496 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:13.174470 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6c263e1-c60a-437a-a13a-579a420a833a-istio-data\") pod \"e2e-shared-a-w4-c34879-openshift-default-ccddfffff-f9pm9\" (UID: \"f6c263e1-c60a-437a-a13a-579a420a833a\") " pod="openshift-ingress/e2e-shared-a-w4-c34879-openshift-default-ccddfffff-f9pm9" Nov 03 02:16:13.174568 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:13.174543 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6c263e1-c60a-437a-a13a-579a420a833a-workload-certs\") pod \"e2e-shared-a-w4-c34879-openshift-default-ccddfffff-f9pm9\" (UID: \"f6c263e1-c60a-437a-a13a-579a420a833a\") " pod="openshift-ingress/e2e-shared-a-w4-c34879-openshift-default-ccddfffff-f9pm9" Nov 03 02:16:13.174627 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:13.174593 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6c263e1-c60a-437a-a13a-579a420a833a-credential-socket\") pod \"e2e-shared-a-w4-c34879-openshift-default-ccddfffff-f9pm9\" (UID: \"f6c263e1-c60a-437a-a13a-579a420a833a\") " pod="openshift-ingress/e2e-shared-a-w4-c34879-openshift-default-ccddfffff-f9pm9" Nov 03 02:16:13.174692 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:13.174633 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6c263e1-c60a-437a-a13a-579a420a833a-workload-socket\") pod \"e2e-shared-a-w4-c34879-openshift-default-ccddfffff-f9pm9\" (UID: \"f6c263e1-c60a-437a-a13a-579a420a833a\") " pod="openshift-ingress/e2e-shared-a-w4-c34879-openshift-default-ccddfffff-f9pm9" Nov 03 02:16:13.174816 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:13.174794 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6c263e1-c60a-437a-a13a-579a420a833a-istio-data\") pod \"e2e-shared-a-w4-c34879-openshift-default-ccddfffff-f9pm9\" (UID: \"f6c263e1-c60a-437a-a13a-579a420a833a\") " pod="openshift-ingress/e2e-shared-a-w4-c34879-openshift-default-ccddfffff-f9pm9" Nov 03 02:16:13.174975 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:13.174956 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6c263e1-c60a-437a-a13a-579a420a833a-istiod-ca-cert\") pod \"e2e-shared-a-w4-c34879-openshift-default-ccddfffff-f9pm9\" (UID: \"f6c263e1-c60a-437a-a13a-579a420a833a\") " pod="openshift-ingress/e2e-shared-a-w4-c34879-openshift-default-ccddfffff-f9pm9" Nov 03 02:16:13.176473 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:13.176450 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6c263e1-c60a-437a-a13a-579a420a833a-istio-envoy\") pod \"e2e-shared-a-w4-c34879-openshift-default-ccddfffff-f9pm9\" (UID: \"f6c263e1-c60a-437a-a13a-579a420a833a\") " pod="openshift-ingress/e2e-shared-a-w4-c34879-openshift-default-ccddfffff-f9pm9" Nov 03 02:16:13.176633 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:13.176612 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6c263e1-c60a-437a-a13a-579a420a833a-istio-podinfo\") pod \"e2e-shared-a-w4-c34879-openshift-default-ccddfffff-f9pm9\" (UID: \"f6c263e1-c60a-437a-a13a-579a420a833a\") " pod="openshift-ingress/e2e-shared-a-w4-c34879-openshift-default-ccddfffff-f9pm9" Nov 03 02:16:13.237537 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:13.237460 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6c263e1-c60a-437a-a13a-579a420a833a-istio-token\") pod \"e2e-shared-a-w4-c34879-openshift-default-ccddfffff-f9pm9\" (UID: \"f6c263e1-c60a-437a-a13a-579a420a833a\") " pod="openshift-ingress/e2e-shared-a-w4-c34879-openshift-default-ccddfffff-f9pm9" Nov 03 02:16:13.237537 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:13.237524 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rlgf\" (UniqueName: \"kubernetes.io/projected/f6c263e1-c60a-437a-a13a-579a420a833a-kube-api-access-4rlgf\") pod \"e2e-shared-a-w4-c34879-openshift-default-ccddfffff-f9pm9\" (UID: \"f6c263e1-c60a-437a-a13a-579a420a833a\") " pod="openshift-ingress/e2e-shared-a-w4-c34879-openshift-default-ccddfffff-f9pm9" Nov 03 02:16:13.258283 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:13.258246 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-c34879-openshift-default-ccddfffff-f9pm9" Nov 03 02:16:13.686166 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:13.686131 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-c34879-openshift-default-ccddfffff-f9pm9"] Nov 03 02:16:13.715823 ip-10-0-136-16 kubenswrapper[5437]: W1103 02:16:13.715792 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6c263e1_c60a_437a_a13a_579a420a833a.slice/crio-ec7f323aff081adee6a12169746f35040d72c9bfff2b2bd81df835d73883776d WatchSource:0}: Error finding container ec7f323aff081adee6a12169746f35040d72c9bfff2b2bd81df835d73883776d: Status 404 returned error can't find the container with id ec7f323aff081adee6a12169746f35040d72c9bfff2b2bd81df835d73883776d Nov 03 02:16:13.718541 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:13.718505 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 02:16:13.718635 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:13.718578 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 02:16:13.718635 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:13.718610 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 02:16:13.875656 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:13.875624 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-c34879-openshift-default-ccddfffff-f9pm9" event={"ID":"f6c263e1-c60a-437a-a13a-579a420a833a","Type":"ContainerStarted","Data":"c023476daba1e99bcd170afb332c31e619826fccb9e6869a06b27bd3583fbf88"} Nov 03 02:16:13.875835 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:13.875681 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-c34879-openshift-default-ccddfffff-f9pm9" event={"ID":"f6c263e1-c60a-437a-a13a-579a420a833a","Type":"ContainerStarted","Data":"ec7f323aff081adee6a12169746f35040d72c9bfff2b2bd81df835d73883776d"} Nov 03 02:16:13.903212 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:13.903171 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:16:14.003072 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:14.002954 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-c34879-openshift-default-ccddfffff-f9pm9" podStartSLOduration=2.002839678 podStartE2EDuration="2.002839678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:16:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:16:14.00020112 +0000 UTC m=+3456.029683436" watchObservedRunningTime="2026-11-03 02:16:14.002839678 +0000 UTC m=+3456.032321976" Nov 03 02:16:14.036732 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:16:14.036692 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 02:16:14.125672 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:14.125629 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:16:14.252054 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:14.252020 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-aab75e-d78bd8b87-5wws6"] Nov 03 02:16:14.256242 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:14.256184 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-aab75e-d78bd8b87-5wws6" Nov 03 02:16:14.258534 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:14.258511 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-c34879-openshift-default-ccddfffff-f9pm9" Nov 03 02:16:14.260111 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:14.260084 5437 patch_prober.go:29] interesting pod/e2e-shared-a-w4-c34879-openshift-default-ccddfffff-f9pm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Nov 03 02:16:14.260214 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:14.260128 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-c34879-openshift-default-ccddfffff-f9pm9" podUID="f6c263e1-c60a-437a-a13a-579a420a833a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Nov 03 02:16:14.324285 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:14.324240 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-aab75e-d78bd8b87-5wws6"] Nov 03 02:16:14.324752 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:14.324725 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-aab75e-dockercfg-9ccsh\"" Nov 03 02:16:14.324953 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:14.324834 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-aab75e\"" Nov 03 02:16:14.392416 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:14.389299 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6687078-b2c2-4a89-9bc9-d0fa069ce917-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-aab75e-d78bd8b87-5wws6\" (UID: \"b6687078-b2c2-4a89-9bc9-d0fa069ce917\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-aab75e-d78bd8b87-5wws6" Nov 03 02:16:14.392416 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:14.389427 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8vmr\" (UniqueName: \"kubernetes.io/projected/b6687078-b2c2-4a89-9bc9-d0fa069ce917-kube-api-access-r8vmr\") pod \"payload-pre-processing-e2e-extmod-a-aab75e-d78bd8b87-5wws6\" (UID: \"b6687078-b2c2-4a89-9bc9-d0fa069ce917\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-aab75e-d78bd8b87-5wws6" Nov 03 02:16:14.396350 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:14.396309 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-aab75e-94fdc78bf-x7ztb"] Nov 03 02:16:14.400627 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:14.400603 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-aab75e-94fdc78bf-x7ztb" Nov 03 02:16:14.454799 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:14.454767 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-aab75e-94fdc78bf-x7ztb"] Nov 03 02:16:14.491547 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:14.491495 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcrfx\" (UniqueName: \"kubernetes.io/projected/97251b61-207f-4a1a-ad2c-89e111f4c38d-kube-api-access-mcrfx\") pod \"payload-processing-e2e-extmod-a-aab75e-94fdc78bf-x7ztb\" (UID: \"97251b61-207f-4a1a-ad2c-89e111f4c38d\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-aab75e-94fdc78bf-x7ztb" Nov 03 02:16:14.491773 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:14.491587 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6687078-b2c2-4a89-9bc9-d0fa069ce917-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-aab75e-d78bd8b87-5wws6\" (UID: \"b6687078-b2c2-4a89-9bc9-d0fa069ce917\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-aab75e-d78bd8b87-5wws6" Nov 03 02:16:14.491773 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:14.491632 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8vmr\" (UniqueName: \"kubernetes.io/projected/b6687078-b2c2-4a89-9bc9-d0fa069ce917-kube-api-access-r8vmr\") pod \"payload-pre-processing-e2e-extmod-a-aab75e-d78bd8b87-5wws6\" (UID: \"b6687078-b2c2-4a89-9bc9-d0fa069ce917\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-aab75e-d78bd8b87-5wws6" Nov 03 02:16:14.491898 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:14.491783 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97251b61-207f-4a1a-ad2c-89e111f4c38d-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-aab75e-94fdc78bf-x7ztb\" (UID: \"97251b61-207f-4a1a-ad2c-89e111f4c38d\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-aab75e-94fdc78bf-x7ztb" Nov 03 02:16:14.492850 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:14.492825 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6687078-b2c2-4a89-9bc9-d0fa069ce917-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-aab75e-d78bd8b87-5wws6\" (UID: \"b6687078-b2c2-4a89-9bc9-d0fa069ce917\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-aab75e-d78bd8b87-5wws6" Nov 03 02:16:14.493406 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:14.493349 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:16:14.564417 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:14.563071 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8vmr\" (UniqueName: \"kubernetes.io/projected/b6687078-b2c2-4a89-9bc9-d0fa069ce917-kube-api-access-r8vmr\") pod \"payload-pre-processing-e2e-extmod-a-aab75e-d78bd8b87-5wws6\" (UID: \"b6687078-b2c2-4a89-9bc9-d0fa069ce917\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-aab75e-d78bd8b87-5wws6" Nov 03 02:16:14.572077 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:14.572046 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-aab75e-d78bd8b87-5wws6" Nov 03 02:16:14.592807 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:14.592776 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97251b61-207f-4a1a-ad2c-89e111f4c38d-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-aab75e-94fdc78bf-x7ztb\" (UID: \"97251b61-207f-4a1a-ad2c-89e111f4c38d\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-aab75e-94fdc78bf-x7ztb" Nov 03 02:16:14.592943 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:14.592889 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcrfx\" (UniqueName: \"kubernetes.io/projected/97251b61-207f-4a1a-ad2c-89e111f4c38d-kube-api-access-mcrfx\") pod \"payload-processing-e2e-extmod-a-aab75e-94fdc78bf-x7ztb\" (UID: \"97251b61-207f-4a1a-ad2c-89e111f4c38d\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-aab75e-94fdc78bf-x7ztb" Nov 03 02:16:14.593788 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:14.593758 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97251b61-207f-4a1a-ad2c-89e111f4c38d-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-aab75e-94fdc78bf-x7ztb\" (UID: \"97251b61-207f-4a1a-ad2c-89e111f4c38d\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-aab75e-94fdc78bf-x7ztb" Nov 03 02:16:14.657860 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:14.657599 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcrfx\" (UniqueName: \"kubernetes.io/projected/97251b61-207f-4a1a-ad2c-89e111f4c38d-kube-api-access-mcrfx\") pod \"payload-processing-e2e-extmod-a-aab75e-94fdc78bf-x7ztb\" (UID: \"97251b61-207f-4a1a-ad2c-89e111f4c38d\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-aab75e-94fdc78bf-x7ztb" Nov 03 02:16:14.715259 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:14.715222 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-aab75e-94fdc78bf-x7ztb" Nov 03 02:16:14.960004 ip-10-0-136-16 kubenswrapper[5437]: W1103 02:16:14.959968 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6687078_b2c2_4a89_9bc9_d0fa069ce917.slice/crio-a96195fa3e586a06d3f5637007d48d4ae8549e8dfd1b18fde94cb641b711a7ce WatchSource:0}: Error finding container a96195fa3e586a06d3f5637007d48d4ae8549e8dfd1b18fde94cb641b711a7ce: Status 404 returned error can't find the container with id a96195fa3e586a06d3f5637007d48d4ae8549e8dfd1b18fde94cb641b711a7ce Nov 03 02:16:14.963041 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:14.962993 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-aab75e-d78bd8b87-5wws6"] Nov 03 02:16:15.140392 ip-10-0-136-16 kubenswrapper[5437]: W1103 02:16:15.129767 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97251b61_207f_4a1a_ad2c_89e111f4c38d.slice/crio-3ccde67baf79e943b783f39817a7d152b0ab4770a3b50e71ddb9f496717be097 WatchSource:0}: Error finding container 3ccde67baf79e943b783f39817a7d152b0ab4770a3b50e71ddb9f496717be097: Status 404 returned error can't find the container with id 3ccde67baf79e943b783f39817a7d152b0ab4770a3b50e71ddb9f496717be097 Nov 03 02:16:15.148391 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:15.144202 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-aab75e-94fdc78bf-x7ztb"] Nov 03 02:16:15.258570 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:15.258545 5437 patch_prober.go:29] interesting pod/e2e-shared-a-w4-c34879-openshift-default-ccddfffff-f9pm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Nov 03 02:16:15.258663 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:15.258595 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-c34879-openshift-default-ccddfffff-f9pm9" podUID="f6c263e1-c60a-437a-a13a-579a420a833a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Nov 03 02:16:15.890177 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:15.890143 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-aab75e-94fdc78bf-x7ztb" event={"ID":"97251b61-207f-4a1a-ad2c-89e111f4c38d","Type":"ContainerStarted","Data":"8ea3a0aa8cf8dbbe71336fe4eff556db264abcdd34c488e8abf064b00159a691"} Nov 03 02:16:15.890395 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:15.890196 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-aab75e-94fdc78bf-x7ztb" event={"ID":"97251b61-207f-4a1a-ad2c-89e111f4c38d","Type":"ContainerStarted","Data":"3ccde67baf79e943b783f39817a7d152b0ab4770a3b50e71ddb9f496717be097"} Nov 03 02:16:15.890395 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:15.890243 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-aab75e-94fdc78bf-x7ztb" Nov 03 02:16:15.892016 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:15.891989 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-aab75e-d78bd8b87-5wws6" event={"ID":"b6687078-b2c2-4a89-9bc9-d0fa069ce917","Type":"ContainerStarted","Data":"594efcb70140b4fa8c9afd8d4ae97e8226dca6038103ab7a0b059053ed536524"} Nov 03 02:16:15.892135 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:15.892036 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-aab75e-d78bd8b87-5wws6" event={"ID":"b6687078-b2c2-4a89-9bc9-d0fa069ce917","Type":"ContainerStarted","Data":"a96195fa3e586a06d3f5637007d48d4ae8549e8dfd1b18fde94cb641b711a7ce"} Nov 03 02:16:15.892135 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:15.892114 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-aab75e-d78bd8b87-5wws6" Nov 03 02:16:16.012028 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:16.011974 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-aab75e-94fdc78bf-x7ztb" podStartSLOduration=2.011959516 podStartE2EDuration="2.011959516s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:16:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:16:16.009338128 +0000 UTC m=+3458.038820435" watchObservedRunningTime="2026-11-03 02:16:16.011959516 +0000 UTC m=+3458.041441811" Nov 03 02:16:16.136389 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:16.134971 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-aab75e-d78bd8b87-5wws6" podStartSLOduration=2.134951911 podStartE2EDuration="2.134951911s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:16:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:16:16.133654229 +0000 UTC m=+3458.163136525" watchObservedRunningTime="2026-11-03 02:16:16.134951911 +0000 UTC m=+3458.164434206" Nov 03 02:16:16.258887 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:16.258800 5437 patch_prober.go:29] interesting pod/e2e-shared-a-w4-c34879-openshift-default-ccddfffff-f9pm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Nov 03 02:16:16.258887 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:16.258860 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-c34879-openshift-default-ccddfffff-f9pm9" podUID="f6c263e1-c60a-437a-a13a-579a420a833a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Nov 03 02:16:17.258800 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:17.258758 5437 patch_prober.go:29] interesting pod/e2e-shared-a-w4-c34879-openshift-default-ccddfffff-f9pm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Nov 03 02:16:17.259000 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:17.258818 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-c34879-openshift-default-ccddfffff-f9pm9" podUID="f6c263e1-c60a-437a-a13a-579a420a833a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Nov 03 02:16:18.212733 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:18.212694 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:16:18.259395 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:18.259340 5437 patch_prober.go:29] interesting pod/e2e-shared-a-w4-c34879-openshift-default-ccddfffff-f9pm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Nov 03 02:16:18.259913 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:18.259437 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-c34879-openshift-default-ccddfffff-f9pm9" podUID="f6c263e1-c60a-437a-a13a-579a420a833a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Nov 03 02:16:18.859718 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:18.859688 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:16:19.259117 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:19.259021 5437 patch_prober.go:29] interesting pod/e2e-shared-a-w4-c34879-openshift-default-ccddfffff-f9pm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Nov 03 02:16:19.259117 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:19.259094 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-c34879-openshift-default-ccddfffff-f9pm9" podUID="f6c263e1-c60a-437a-a13a-579a420a833a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Nov 03 02:16:19.311987 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:19.311947 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-50ca6d-openshift-default-6b7b98d8c7-499js"] Nov 03 02:16:19.315192 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:19.315161 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:16:19.316622 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:19.316604 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-50ca6d-openshift-default-6b7b98d8c7-499js" Nov 03 02:16:19.371240 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:19.371206 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-50ca6d-openshift-default-dockercfg-m7fzs\"" Nov 03 02:16:19.374453 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:19.374423 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-50ca6d-openshift-default-6b7b98d8c7-499js"] Nov 03 02:16:19.474613 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:19.474582 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea-istio-data\") pod \"e2e-autores-50ca6d-openshift-default-6b7b98d8c7-499js\" (UID: \"6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea\") " pod="openshift-ingress/e2e-autores-50ca6d-openshift-default-6b7b98d8c7-499js" Nov 03 02:16:19.474613 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:19.474626 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea-workload-socket\") pod \"e2e-autores-50ca6d-openshift-default-6b7b98d8c7-499js\" (UID: \"6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea\") " pod="openshift-ingress/e2e-autores-50ca6d-openshift-default-6b7b98d8c7-499js" Nov 03 02:16:19.474854 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:19.474755 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea-workload-certs\") pod \"e2e-autores-50ca6d-openshift-default-6b7b98d8c7-499js\" (UID: \"6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea\") " pod="openshift-ingress/e2e-autores-50ca6d-openshift-default-6b7b98d8c7-499js" Nov 03 02:16:19.474854 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:19.474771 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea-istio-envoy\") pod \"e2e-autores-50ca6d-openshift-default-6b7b98d8c7-499js\" (UID: \"6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea\") " pod="openshift-ingress/e2e-autores-50ca6d-openshift-default-6b7b98d8c7-499js" Nov 03 02:16:19.474932 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:19.474907 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea-istiod-ca-cert\") pod \"e2e-autores-50ca6d-openshift-default-6b7b98d8c7-499js\" (UID: \"6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea\") " pod="openshift-ingress/e2e-autores-50ca6d-openshift-default-6b7b98d8c7-499js" Nov 03 02:16:19.474971 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:19.474959 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea-istio-token\") pod \"e2e-autores-50ca6d-openshift-default-6b7b98d8c7-499js\" (UID: \"6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea\") " pod="openshift-ingress/e2e-autores-50ca6d-openshift-default-6b7b98d8c7-499js" Nov 03 02:16:19.475011 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:19.474989 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea-istio-podinfo\") pod \"e2e-autores-50ca6d-openshift-default-6b7b98d8c7-499js\" (UID: \"6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea\") " pod="openshift-ingress/e2e-autores-50ca6d-openshift-default-6b7b98d8c7-499js" Nov 03 02:16:19.475066 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:19.475045 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnmb4\" (UniqueName: \"kubernetes.io/projected/6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea-kube-api-access-bnmb4\") pod \"e2e-autores-50ca6d-openshift-default-6b7b98d8c7-499js\" (UID: \"6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea\") " pod="openshift-ingress/e2e-autores-50ca6d-openshift-default-6b7b98d8c7-499js" Nov 03 02:16:19.475121 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:19.475084 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea-credential-socket\") pod \"e2e-autores-50ca6d-openshift-default-6b7b98d8c7-499js\" (UID: \"6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea\") " pod="openshift-ingress/e2e-autores-50ca6d-openshift-default-6b7b98d8c7-499js" Nov 03 02:16:19.540449 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:19.539922 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:16:19.576788 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:19.576753 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea-workload-certs\") pod \"e2e-autores-50ca6d-openshift-default-6b7b98d8c7-499js\" (UID: \"6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea\") " pod="openshift-ingress/e2e-autores-50ca6d-openshift-default-6b7b98d8c7-499js" Nov 03 02:16:19.576788 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:19.576788 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea-istio-envoy\") pod \"e2e-autores-50ca6d-openshift-default-6b7b98d8c7-499js\" (UID: \"6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea\") " pod="openshift-ingress/e2e-autores-50ca6d-openshift-default-6b7b98d8c7-499js" Nov 03 02:16:19.577015 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:19.576848 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea-istiod-ca-cert\") pod \"e2e-autores-50ca6d-openshift-default-6b7b98d8c7-499js\" (UID: \"6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea\") " pod="openshift-ingress/e2e-autores-50ca6d-openshift-default-6b7b98d8c7-499js" Nov 03 02:16:19.577015 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:19.576883 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea-istio-token\") pod \"e2e-autores-50ca6d-openshift-default-6b7b98d8c7-499js\" (UID: \"6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea\") " pod="openshift-ingress/e2e-autores-50ca6d-openshift-default-6b7b98d8c7-499js" Nov 03 02:16:19.577015 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:19.576911 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea-istio-podinfo\") pod \"e2e-autores-50ca6d-openshift-default-6b7b98d8c7-499js\" (UID: \"6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea\") " pod="openshift-ingress/e2e-autores-50ca6d-openshift-default-6b7b98d8c7-499js" Nov 03 02:16:19.577015 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:19.576967 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnmb4\" (UniqueName: \"kubernetes.io/projected/6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea-kube-api-access-bnmb4\") pod \"e2e-autores-50ca6d-openshift-default-6b7b98d8c7-499js\" (UID: \"6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea\") " pod="openshift-ingress/e2e-autores-50ca6d-openshift-default-6b7b98d8c7-499js" Nov 03 02:16:19.577015 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:19.577006 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea-credential-socket\") pod \"e2e-autores-50ca6d-openshift-default-6b7b98d8c7-499js\" (UID: \"6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea\") " pod="openshift-ingress/e2e-autores-50ca6d-openshift-default-6b7b98d8c7-499js" Nov 03 02:16:19.577190 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:19.577072 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea-istio-data\") pod \"e2e-autores-50ca6d-openshift-default-6b7b98d8c7-499js\" (UID: \"6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea\") " pod="openshift-ingress/e2e-autores-50ca6d-openshift-default-6b7b98d8c7-499js" Nov 03 02:16:19.577190 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:19.577123 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea-workload-socket\") pod \"e2e-autores-50ca6d-openshift-default-6b7b98d8c7-499js\" (UID: \"6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea\") " pod="openshift-ingress/e2e-autores-50ca6d-openshift-default-6b7b98d8c7-499js" Nov 03 02:16:19.577261 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:19.577209 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea-workload-certs\") pod \"e2e-autores-50ca6d-openshift-default-6b7b98d8c7-499js\" (UID: \"6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea\") " pod="openshift-ingress/e2e-autores-50ca6d-openshift-default-6b7b98d8c7-499js" Nov 03 02:16:19.577448 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:19.577426 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea-credential-socket\") pod \"e2e-autores-50ca6d-openshift-default-6b7b98d8c7-499js\" (UID: \"6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea\") " pod="openshift-ingress/e2e-autores-50ca6d-openshift-default-6b7b98d8c7-499js" Nov 03 02:16:19.577493 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:19.577474 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea-workload-socket\") pod \"e2e-autores-50ca6d-openshift-default-6b7b98d8c7-499js\" (UID: \"6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea\") " pod="openshift-ingress/e2e-autores-50ca6d-openshift-default-6b7b98d8c7-499js" Nov 03 02:16:19.577637 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:19.577620 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea-istio-data\") pod \"e2e-autores-50ca6d-openshift-default-6b7b98d8c7-499js\" (UID: \"6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea\") " pod="openshift-ingress/e2e-autores-50ca6d-openshift-default-6b7b98d8c7-499js" Nov 03 02:16:19.578103 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:19.578067 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea-istiod-ca-cert\") pod \"e2e-autores-50ca6d-openshift-default-6b7b98d8c7-499js\" (UID: \"6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea\") " pod="openshift-ingress/e2e-autores-50ca6d-openshift-default-6b7b98d8c7-499js" Nov 03 02:16:19.579700 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:19.579674 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea-istio-envoy\") pod \"e2e-autores-50ca6d-openshift-default-6b7b98d8c7-499js\" (UID: \"6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea\") " pod="openshift-ingress/e2e-autores-50ca6d-openshift-default-6b7b98d8c7-499js" Nov 03 02:16:19.581780 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:19.581755 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea-istio-podinfo\") pod \"e2e-autores-50ca6d-openshift-default-6b7b98d8c7-499js\" (UID: \"6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea\") " pod="openshift-ingress/e2e-autores-50ca6d-openshift-default-6b7b98d8c7-499js" Nov 03 02:16:19.644116 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:19.644079 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnmb4\" (UniqueName: \"kubernetes.io/projected/6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea-kube-api-access-bnmb4\") pod \"e2e-autores-50ca6d-openshift-default-6b7b98d8c7-499js\" (UID: \"6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea\") " pod="openshift-ingress/e2e-autores-50ca6d-openshift-default-6b7b98d8c7-499js" Nov 03 02:16:19.644892 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:19.644868 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea-istio-token\") pod \"e2e-autores-50ca6d-openshift-default-6b7b98d8c7-499js\" (UID: \"6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea\") " pod="openshift-ingress/e2e-autores-50ca6d-openshift-default-6b7b98d8c7-499js" Nov 03 02:16:19.655919 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:19.655887 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-50ca6d-openshift-default-6b7b98d8c7-499js" Nov 03 02:16:19.747765 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:19.747715 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:16:20.097118 ip-10-0-136-16 kubenswrapper[5437]: W1103 02:16:20.097081 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cc0aaa6_f284_44a2_a2c8_bb0283ce68ea.slice/crio-ab5acd4584d51e3da66baa13d78435cd5e834185015e5a7038e364f7139ed420 WatchSource:0}: Error finding container ab5acd4584d51e3da66baa13d78435cd5e834185015e5a7038e364f7139ed420: Status 404 returned error can't find the container with id ab5acd4584d51e3da66baa13d78435cd5e834185015e5a7038e364f7139ed420 Nov 03 02:16:20.099684 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:20.099660 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-50ca6d-openshift-default-6b7b98d8c7-499js"] Nov 03 02:16:20.099862 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:20.099835 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 02:16:20.099910 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:20.099898 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 02:16:20.099947 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:20.099928 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 02:16:20.259274 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:20.259232 5437 patch_prober.go:29] interesting pod/e2e-shared-a-w4-c34879-openshift-default-ccddfffff-f9pm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Nov 03 02:16:20.259550 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:20.259298 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-c34879-openshift-default-ccddfffff-f9pm9" podUID="f6c263e1-c60a-437a-a13a-579a420a833a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Nov 03 02:16:20.925091 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:20.925054 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-50ca6d-openshift-default-6b7b98d8c7-499js" event={"ID":"6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea","Type":"ContainerStarted","Data":"659c9112015d4f3ab0715fd52664d159b8a66d7dc8ba8c91c0ff6a85772096b4"} Nov 03 02:16:20.925601 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:20.925106 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-50ca6d-openshift-default-6b7b98d8c7-499js" event={"ID":"6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea","Type":"ContainerStarted","Data":"ab5acd4584d51e3da66baa13d78435cd5e834185015e5a7038e364f7139ed420"} Nov 03 02:16:21.057144 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:21.057086 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-50ca6d-openshift-default-6b7b98d8c7-499js" podStartSLOduration=2.057072891 podStartE2EDuration="2.057072891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:16:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:16:21.054623909 +0000 UTC m=+3463.084106204" watchObservedRunningTime="2026-11-03 02:16:21.057072891 +0000 UTC m=+3463.086555185" Nov 03 02:16:21.264021 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:21.263942 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-c34879-openshift-default-ccddfffff-f9pm9" Nov 03 02:16:21.264252 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:21.264233 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-c34879-openshift-default-ccddfffff-f9pm9" Nov 03 02:16:21.265018 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:21.264998 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-c34879-openshift-default-ccddfffff-f9pm9" Nov 03 02:16:21.656912 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:21.656870 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-50ca6d-openshift-default-6b7b98d8c7-499js" Nov 03 02:16:21.663619 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:21.663593 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-50ca6d-openshift-default-6b7b98d8c7-499js" Nov 03 02:16:21.931431 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:21.931319 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-50ca6d-openshift-default-6b7b98d8c7-499js" Nov 03 02:16:21.932181 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:21.932164 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-50ca6d-openshift-default-6b7b98d8c7-499js" Nov 03 02:16:23.906240 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:23.906199 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:16:24.125199 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:24.125165 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:16:24.328088 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:24.328049 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:16:26.901469 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:26.901433 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-aab75e-d78bd8b87-5wws6" Nov 03 02:16:26.901963 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:26.901658 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-aab75e-94fdc78bf-x7ztb" Nov 03 02:16:27.005256 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:27.005223 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:16:27.212597 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:27.212503 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:16:27.517453 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:27.517337 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-50ca6d-774c6bf95-b26nc"] Nov 03 02:16:27.522616 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:27.522590 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-50ca6d-774c6bf95-b26nc" Nov 03 02:16:27.575028 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:27.574997 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-50ca6d-774c6bf95-b26nc"] Nov 03 02:16:27.590240 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:27.590198 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50d03554-8ebc-498c-848f-0f0dbc24dd41-plugins-config-volume\") pod \"payload-processing-e2e-autores-50ca6d-774c6bf95-b26nc\" (UID: \"50d03554-8ebc-498c-848f-0f0dbc24dd41\") " pod="openshift-ingress/payload-processing-e2e-autores-50ca6d-774c6bf95-b26nc" Nov 03 02:16:27.590473 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:27.590271 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgqm6\" (UniqueName: \"kubernetes.io/projected/50d03554-8ebc-498c-848f-0f0dbc24dd41-kube-api-access-bgqm6\") pod \"payload-processing-e2e-autores-50ca6d-774c6bf95-b26nc\" (UID: \"50d03554-8ebc-498c-848f-0f0dbc24dd41\") " pod="openshift-ingress/payload-processing-e2e-autores-50ca6d-774c6bf95-b26nc" Nov 03 02:16:27.624003 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:27.623972 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-50ca6d\"" Nov 03 02:16:27.624003 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:27.623973 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-50ca6d-dockercfg-h7fds\"" Nov 03 02:16:27.691674 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:27.691619 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50d03554-8ebc-498c-848f-0f0dbc24dd41-plugins-config-volume\") pod \"payload-processing-e2e-autores-50ca6d-774c6bf95-b26nc\" (UID: \"50d03554-8ebc-498c-848f-0f0dbc24dd41\") " pod="openshift-ingress/payload-processing-e2e-autores-50ca6d-774c6bf95-b26nc" Nov 03 02:16:27.691896 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:27.691734 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgqm6\" (UniqueName: \"kubernetes.io/projected/50d03554-8ebc-498c-848f-0f0dbc24dd41-kube-api-access-bgqm6\") pod \"payload-processing-e2e-autores-50ca6d-774c6bf95-b26nc\" (UID: \"50d03554-8ebc-498c-848f-0f0dbc24dd41\") " pod="openshift-ingress/payload-processing-e2e-autores-50ca6d-774c6bf95-b26nc" Nov 03 02:16:27.692502 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:27.692480 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50d03554-8ebc-498c-848f-0f0dbc24dd41-plugins-config-volume\") pod \"payload-processing-e2e-autores-50ca6d-774c6bf95-b26nc\" (UID: \"50d03554-8ebc-498c-848f-0f0dbc24dd41\") " pod="openshift-ingress/payload-processing-e2e-autores-50ca6d-774c6bf95-b26nc" Nov 03 02:16:27.755413 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:27.755355 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgqm6\" (UniqueName: \"kubernetes.io/projected/50d03554-8ebc-498c-848f-0f0dbc24dd41-kube-api-access-bgqm6\") pod \"payload-processing-e2e-autores-50ca6d-774c6bf95-b26nc\" (UID: \"50d03554-8ebc-498c-848f-0f0dbc24dd41\") " pod="openshift-ingress/payload-processing-e2e-autores-50ca6d-774c6bf95-b26nc" Nov 03 02:16:27.839659 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:27.839627 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-50ca6d-774c6bf95-b26nc" Nov 03 02:16:28.220055 ip-10-0-136-16 kubenswrapper[5437]: W1103 02:16:28.220023 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50d03554_8ebc_498c_848f_0f0dbc24dd41.slice/crio-085f224b18575b19cc76da3186c67580a1f1416cae1db59db3c27677fff150c2 WatchSource:0}: Error finding container 085f224b18575b19cc76da3186c67580a1f1416cae1db59db3c27677fff150c2: Status 404 returned error can't find the container with id 085f224b18575b19cc76da3186c67580a1f1416cae1db59db3c27677fff150c2 Nov 03 02:16:28.224063 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:28.224032 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-50ca6d-774c6bf95-b26nc"] Nov 03 02:16:28.976813 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:28.976774 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-50ca6d-774c6bf95-b26nc" event={"ID":"50d03554-8ebc-498c-848f-0f0dbc24dd41","Type":"ContainerStarted","Data":"0183c1b9d6dbe402818660d32b905f2435403220dc6a2919210619070e3592d9"} Nov 03 02:16:28.977026 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:28.976840 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-50ca6d-774c6bf95-b26nc" event={"ID":"50d03554-8ebc-498c-848f-0f0dbc24dd41","Type":"ContainerStarted","Data":"085f224b18575b19cc76da3186c67580a1f1416cae1db59db3c27677fff150c2"} Nov 03 02:16:28.977026 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:28.976981 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-50ca6d-774c6bf95-b26nc" Nov 03 02:16:29.095928 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:29.095874 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-50ca6d-774c6bf95-b26nc" podStartSLOduration=2.095860366 podStartE2EDuration="2.095860366s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:16:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:16:29.095580993 +0000 UTC m=+3471.125063289" watchObservedRunningTime="2026-11-03 02:16:29.095860366 +0000 UTC m=+3471.125342660" Nov 03 02:16:29.318133 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:29.318094 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:16:31.090629 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:31.090584 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:16:31.702785 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:31.702745 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:16:31.957053 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:31.956954 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:16:34.512313 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:34.512271 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:16:35.870041 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:35.870002 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:16:36.077110 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:36.077068 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:16:39.054609 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:39.054568 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:16:39.265968 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:39.265933 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:16:39.577271 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:39.577236 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:16:39.777632 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:39.777586 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:16:39.996055 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:39.995967 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-50ca6d-774c6bf95-b26nc" Nov 03 02:16:40.540910 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:40.540869 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:16:40.779536 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:40.779491 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:16:44.035692 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:16:44.035652 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 02:16:44.792780 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:44.792740 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:16:45.824653 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:45.824612 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:16:49.627799 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:49.627765 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:16:54.391755 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:54.391713 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:16:54.651932 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:54.651848 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:16:55.654749 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:55.654710 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:16:58.577944 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:58.577894 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:16:59.309287 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:16:59.309252 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:17:00.287305 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:17:00.287271 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:17:00.513539 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:17:00.513498 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:17:01.773672 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:17:01.773632 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:17:02.288927 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:17:02.288888 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:17:02.961253 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:17:02.961221 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:17:03.593778 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:17:03.593741 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:17:03.854893 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:17:03.854793 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:17:05.211358 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:17:05.211317 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:17:05.635252 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:17:05.635219 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:17:05.841977 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:17:05.841944 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:17:06.047238 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:17:06.047199 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:17:10.090408 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:17:10.090345 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:17:14.034836 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:17:14.034798 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 02:17:14.872488 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:17:14.872436 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:17:19.607129 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:17:19.607041 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:17:20.973680 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:17:20.973637 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:17:22.286535 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:17:22.286500 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:17:23.912007 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:17:23.911968 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:17:24.120189 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:17:24.120139 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:17:24.633603 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:17:24.633547 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:17:24.885024 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:17:24.884939 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:17:28.720301 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:17:28.720264 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:17:28.975624 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:17:28.975523 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:17:30.002991 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:17:30.002952 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:17:34.114032 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:17:34.113999 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:17:34.318834 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:17:34.318783 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:17:35.134130 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:17:35.134087 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:17:40.215953 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:17:40.215904 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:17:44.034415 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:17:44.034378 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 02:17:44.851653 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:17:44.851623 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:17:45.056620 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:17:45.056581 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:17:49.875457 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:17:49.875420 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:17:54.644079 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:17:54.644027 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:17:56.006427 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:17:56.006391 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:17:59.072833 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:17:59.072797 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-d8cc55-6f4d7f5cf4-njhfg"] Nov 03 02:17:59.077190 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:17:59.077170 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d8cc55-6f4d7f5cf4-njhfg" Nov 03 02:17:59.131730 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:17:59.131697 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-d8cc55-6f4d7f5cf4-njhfg"] Nov 03 02:17:59.180514 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:17:59.180478 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-d8cc55-serving-cert\"" Nov 03 02:17:59.226595 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:17:59.226562 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brdzj\" (UniqueName: \"kubernetes.io/projected/8721e215-270a-4092-b0ef-16fb74a4b863-kube-api-access-brdzj\") pod \"maas-api-e2e-extmod-a-d8cc55-6f4d7f5cf4-njhfg\" (UID: \"8721e215-270a-4092-b0ef-16fb74a4b863\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d8cc55-6f4d7f5cf4-njhfg" Nov 03 02:17:59.226786 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:17:59.226762 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8721e215-270a-4092-b0ef-16fb74a4b863-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-d8cc55-6f4d7f5cf4-njhfg\" (UID: \"8721e215-270a-4092-b0ef-16fb74a4b863\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d8cc55-6f4d7f5cf4-njhfg" Nov 03 02:17:59.227221 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:17:59.227186 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8721e215-270a-4092-b0ef-16fb74a4b863-maas-api-tls\") pod \"maas-api-e2e-extmod-a-d8cc55-6f4d7f5cf4-njhfg\" (UID: \"8721e215-270a-4092-b0ef-16fb74a4b863\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d8cc55-6f4d7f5cf4-njhfg" Nov 03 02:17:59.227392 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:17:59.227231 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8721e215-270a-4092-b0ef-16fb74a4b863-metrics-certs\") pod \"maas-api-e2e-extmod-a-d8cc55-6f4d7f5cf4-njhfg\" (UID: \"8721e215-270a-4092-b0ef-16fb74a4b863\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d8cc55-6f4d7f5cf4-njhfg" Nov 03 02:17:59.311242 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:17:59.311205 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:17:59.328412 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:17:59.328292 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8721e215-270a-4092-b0ef-16fb74a4b863-maas-api-tls\") pod \"maas-api-e2e-extmod-a-d8cc55-6f4d7f5cf4-njhfg\" (UID: \"8721e215-270a-4092-b0ef-16fb74a4b863\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d8cc55-6f4d7f5cf4-njhfg" Nov 03 02:17:59.328412 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:17:59.328354 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8721e215-270a-4092-b0ef-16fb74a4b863-metrics-certs\") pod \"maas-api-e2e-extmod-a-d8cc55-6f4d7f5cf4-njhfg\" (UID: \"8721e215-270a-4092-b0ef-16fb74a4b863\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d8cc55-6f4d7f5cf4-njhfg" Nov 03 02:17:59.328656 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:17:59.328410 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brdzj\" (UniqueName: \"kubernetes.io/projected/8721e215-270a-4092-b0ef-16fb74a4b863-kube-api-access-brdzj\") pod \"maas-api-e2e-extmod-a-d8cc55-6f4d7f5cf4-njhfg\" (UID: \"8721e215-270a-4092-b0ef-16fb74a4b863\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d8cc55-6f4d7f5cf4-njhfg" Nov 03 02:17:59.328656 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:17:59.328497 5437 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-d8cc55-serving-cert: secret "maas-api-e2e-extmod-a-d8cc55-serving-cert" not found Nov 03 02:17:59.328656 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:17:59.328533 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8721e215-270a-4092-b0ef-16fb74a4b863-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-d8cc55-6f4d7f5cf4-njhfg\" (UID: \"8721e215-270a-4092-b0ef-16fb74a4b863\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d8cc55-6f4d7f5cf4-njhfg" Nov 03 02:17:59.328656 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:17:59.328576 5437 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8721e215-270a-4092-b0ef-16fb74a4b863-maas-api-tls podName:8721e215-270a-4092-b0ef-16fb74a4b863 nodeName:}" failed. No retries permitted until 2026-11-03 02:17:59.828553964 +0000 UTC m=+3561.858036243 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8721e215-270a-4092-b0ef-16fb74a4b863-maas-api-tls") pod "maas-api-e2e-extmod-a-d8cc55-6f4d7f5cf4-njhfg" (UID: "8721e215-270a-4092-b0ef-16fb74a4b863") : secret "maas-api-e2e-extmod-a-d8cc55-serving-cert" not found Nov 03 02:17:59.329181 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:17:59.329160 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8721e215-270a-4092-b0ef-16fb74a4b863-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-d8cc55-6f4d7f5cf4-njhfg\" (UID: \"8721e215-270a-4092-b0ef-16fb74a4b863\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d8cc55-6f4d7f5cf4-njhfg" Nov 03 02:17:59.330915 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:17:59.330891 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8721e215-270a-4092-b0ef-16fb74a4b863-metrics-certs\") pod \"maas-api-e2e-extmod-a-d8cc55-6f4d7f5cf4-njhfg\" (UID: \"8721e215-270a-4092-b0ef-16fb74a4b863\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d8cc55-6f4d7f5cf4-njhfg" Nov 03 02:17:59.392016 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:17:59.391986 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brdzj\" (UniqueName: \"kubernetes.io/projected/8721e215-270a-4092-b0ef-16fb74a4b863-kube-api-access-brdzj\") pod \"maas-api-e2e-extmod-a-d8cc55-6f4d7f5cf4-njhfg\" (UID: \"8721e215-270a-4092-b0ef-16fb74a4b863\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d8cc55-6f4d7f5cf4-njhfg" Nov 03 02:17:59.566346 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:17:59.566309 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:17:59.768137 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:17:59.768049 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:17:59.834516 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:17:59.834482 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8721e215-270a-4092-b0ef-16fb74a4b863-maas-api-tls\") pod \"maas-api-e2e-extmod-a-d8cc55-6f4d7f5cf4-njhfg\" (UID: \"8721e215-270a-4092-b0ef-16fb74a4b863\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d8cc55-6f4d7f5cf4-njhfg" Nov 03 02:17:59.837053 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:17:59.837028 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8721e215-270a-4092-b0ef-16fb74a4b863-maas-api-tls\") pod \"maas-api-e2e-extmod-a-d8cc55-6f4d7f5cf4-njhfg\" (UID: \"8721e215-270a-4092-b0ef-16fb74a4b863\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d8cc55-6f4d7f5cf4-njhfg" Nov 03 02:17:59.992059 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:17:59.992025 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d8cc55-6f4d7f5cf4-njhfg" Nov 03 02:18:00.041116 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:18:00.041025 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-898926/noten-model-898926-kserve-5b6b95d9bf-glsl2"] Nov 03 02:18:00.047946 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:18:00.045905 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-898926/noten-model-898926-kserve-5b6b95d9bf-glsl2" Nov 03 02:18:00.100646 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:18:00.100614 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-898926/noten-model-898926-kserve-5b6b95d9bf-glsl2"] Nov 03 02:18:00.148176 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:18:00.148145 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-898926\"/\"openshift-service-ca.crt\"" Nov 03 02:18:00.148176 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:18:00.148154 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-898926\"/\"default-dockercfg-dvnmd\"" Nov 03 02:18:00.148432 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:18:00.148224 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-898926\"/\"noten-model-898926-kserve-self-signed-certs\"" Nov 03 02:18:00.152718 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:18:00.152698 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-898926\"/\"kube-root-ca.crt\"" Nov 03 02:18:00.186075 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:18:00.186036 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:18:00.240314 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:18:00.240277 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42963aa8-67e0-49e3-9ae4-29e6545469ac-model-cache\") pod \"noten-model-898926-kserve-5b6b95d9bf-glsl2\" (UID: \"42963aa8-67e0-49e3-9ae4-29e6545469ac\") " pod="ai-tenant-e2e-noten-898926/noten-model-898926-kserve-5b6b95d9bf-glsl2" Nov 03 02:18:00.240673 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:18:00.240360 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42963aa8-67e0-49e3-9ae4-29e6545469ac-dshm\") pod \"noten-model-898926-kserve-5b6b95d9bf-glsl2\" (UID: \"42963aa8-67e0-49e3-9ae4-29e6545469ac\") " pod="ai-tenant-e2e-noten-898926/noten-model-898926-kserve-5b6b95d9bf-glsl2" Nov 03 02:18:00.240673 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:18:00.240472 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42963aa8-67e0-49e3-9ae4-29e6545469ac-tls-certs\") pod \"noten-model-898926-kserve-5b6b95d9bf-glsl2\" (UID: \"42963aa8-67e0-49e3-9ae4-29e6545469ac\") " pod="ai-tenant-e2e-noten-898926/noten-model-898926-kserve-5b6b95d9bf-glsl2" Nov 03 02:18:00.240673 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:18:00.240564 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvl8d\" (UniqueName: \"kubernetes.io/projected/42963aa8-67e0-49e3-9ae4-29e6545469ac-kube-api-access-kvl8d\") pod \"noten-model-898926-kserve-5b6b95d9bf-glsl2\" (UID: \"42963aa8-67e0-49e3-9ae4-29e6545469ac\") " pod="ai-tenant-e2e-noten-898926/noten-model-898926-kserve-5b6b95d9bf-glsl2" Nov 03 02:18:00.240673 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:18:00.240638 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42963aa8-67e0-49e3-9ae4-29e6545469ac-home\") pod \"noten-model-898926-kserve-5b6b95d9bf-glsl2\" (UID: \"42963aa8-67e0-49e3-9ae4-29e6545469ac\") " pod="ai-tenant-e2e-noten-898926/noten-model-898926-kserve-5b6b95d9bf-glsl2" Nov 03 02:18:00.240906 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:18:00.240689 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42963aa8-67e0-49e3-9ae4-29e6545469ac-tmp-dir\") pod \"noten-model-898926-kserve-5b6b95d9bf-glsl2\" (UID: \"42963aa8-67e0-49e3-9ae4-29e6545469ac\") " pod="ai-tenant-e2e-noten-898926/noten-model-898926-kserve-5b6b95d9bf-glsl2" Nov 03 02:18:00.342456 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:18:00.342413 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvl8d\" (UniqueName: \"kubernetes.io/projected/42963aa8-67e0-49e3-9ae4-29e6545469ac-kube-api-access-kvl8d\") pod \"noten-model-898926-kserve-5b6b95d9bf-glsl2\" (UID: \"42963aa8-67e0-49e3-9ae4-29e6545469ac\") " pod="ai-tenant-e2e-noten-898926/noten-model-898926-kserve-5b6b95d9bf-glsl2" Nov 03 02:18:00.342659 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:18:00.342473 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42963aa8-67e0-49e3-9ae4-29e6545469ac-home\") pod \"noten-model-898926-kserve-5b6b95d9bf-glsl2\" (UID: \"42963aa8-67e0-49e3-9ae4-29e6545469ac\") " pod="ai-tenant-e2e-noten-898926/noten-model-898926-kserve-5b6b95d9bf-glsl2" Nov 03 02:18:00.342659 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:18:00.342513 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42963aa8-67e0-49e3-9ae4-29e6545469ac-tmp-dir\") pod \"noten-model-898926-kserve-5b6b95d9bf-glsl2\" (UID: \"42963aa8-67e0-49e3-9ae4-29e6545469ac\") " pod="ai-tenant-e2e-noten-898926/noten-model-898926-kserve-5b6b95d9bf-glsl2" Nov 03 02:18:00.342659 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:18:00.342612 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42963aa8-67e0-49e3-9ae4-29e6545469ac-model-cache\") pod \"noten-model-898926-kserve-5b6b95d9bf-glsl2\" (UID: \"42963aa8-67e0-49e3-9ae4-29e6545469ac\") " pod="ai-tenant-e2e-noten-898926/noten-model-898926-kserve-5b6b95d9bf-glsl2" Nov 03 02:18:00.342830 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:18:00.342692 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42963aa8-67e0-49e3-9ae4-29e6545469ac-dshm\") pod \"noten-model-898926-kserve-5b6b95d9bf-glsl2\" (UID: \"42963aa8-67e0-49e3-9ae4-29e6545469ac\") " pod="ai-tenant-e2e-noten-898926/noten-model-898926-kserve-5b6b95d9bf-glsl2" Nov 03 02:18:00.342830 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:18:00.342799 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42963aa8-67e0-49e3-9ae4-29e6545469ac-tls-certs\") pod \"noten-model-898926-kserve-5b6b95d9bf-glsl2\" (UID: \"42963aa8-67e0-49e3-9ae4-29e6545469ac\") " pod="ai-tenant-e2e-noten-898926/noten-model-898926-kserve-5b6b95d9bf-glsl2" Nov 03 02:18:00.342973 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:18:00.342945 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42963aa8-67e0-49e3-9ae4-29e6545469ac-home\") pod \"noten-model-898926-kserve-5b6b95d9bf-glsl2\" (UID: \"42963aa8-67e0-49e3-9ae4-29e6545469ac\") " pod="ai-tenant-e2e-noten-898926/noten-model-898926-kserve-5b6b95d9bf-glsl2" Nov 03 02:18:00.343096 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:18:00.342980 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42963aa8-67e0-49e3-9ae4-29e6545469ac-tmp-dir\") pod \"noten-model-898926-kserve-5b6b95d9bf-glsl2\" (UID: \"42963aa8-67e0-49e3-9ae4-29e6545469ac\") " pod="ai-tenant-e2e-noten-898926/noten-model-898926-kserve-5b6b95d9bf-glsl2" Nov 03 02:18:00.343096 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:18:00.343044 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42963aa8-67e0-49e3-9ae4-29e6545469ac-model-cache\") pod \"noten-model-898926-kserve-5b6b95d9bf-glsl2\" (UID: \"42963aa8-67e0-49e3-9ae4-29e6545469ac\") " pod="ai-tenant-e2e-noten-898926/noten-model-898926-kserve-5b6b95d9bf-glsl2" Nov 03 02:18:00.344955 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:18:00.344935 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42963aa8-67e0-49e3-9ae4-29e6545469ac-dshm\") pod \"noten-model-898926-kserve-5b6b95d9bf-glsl2\" (UID: \"42963aa8-67e0-49e3-9ae4-29e6545469ac\") " pod="ai-tenant-e2e-noten-898926/noten-model-898926-kserve-5b6b95d9bf-glsl2" Nov 03 02:18:00.345204 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:18:00.345186 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42963aa8-67e0-49e3-9ae4-29e6545469ac-tls-certs\") pod \"noten-model-898926-kserve-5b6b95d9bf-glsl2\" (UID: \"42963aa8-67e0-49e3-9ae4-29e6545469ac\") " pod="ai-tenant-e2e-noten-898926/noten-model-898926-kserve-5b6b95d9bf-glsl2" Nov 03 02:18:00.409393 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:18:00.409334 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvl8d\" (UniqueName: \"kubernetes.io/projected/42963aa8-67e0-49e3-9ae4-29e6545469ac-kube-api-access-kvl8d\") pod \"noten-model-898926-kserve-5b6b95d9bf-glsl2\" (UID: \"42963aa8-67e0-49e3-9ae4-29e6545469ac\") " pod="ai-tenant-e2e-noten-898926/noten-model-898926-kserve-5b6b95d9bf-glsl2" Nov 03 02:18:00.412623 ip-10-0-136-16 kubenswrapper[5437]: W1103 02:18:00.412596 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8721e215_270a_4092_b0ef_16fb74a4b863.slice/crio-c1b3856842e2659599ba1b775caa7379d7c3d6deb20dd46164fa571936efc9d9 WatchSource:0}: Error finding container c1b3856842e2659599ba1b775caa7379d7c3d6deb20dd46164fa571936efc9d9: Status 404 returned error can't find the container with id c1b3856842e2659599ba1b775caa7379d7c3d6deb20dd46164fa571936efc9d9 Nov 03 02:18:00.416407 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:18:00.416345 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-d8cc55-6f4d7f5cf4-njhfg"] Nov 03 02:18:00.579757 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:18:00.579723 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d8cc55-6f4d7f5cf4-njhfg" event={"ID":"8721e215-270a-4092-b0ef-16fb74a4b863","Type":"ContainerStarted","Data":"c1b3856842e2659599ba1b775caa7379d7c3d6deb20dd46164fa571936efc9d9"} Nov 03 02:18:00.677749 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:18:00.677660 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-898926/noten-model-898926-kserve-5b6b95d9bf-glsl2" Nov 03 02:18:00.820753 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:18:00.820717 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:18:01.058649 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:18:01.058613 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-898926/noten-model-898926-kserve-5b6b95d9bf-glsl2"] Nov 03 02:18:01.588223 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:18:01.588128 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-898926/noten-model-898926-kserve-5b6b95d9bf-glsl2" event={"ID":"42963aa8-67e0-49e3-9ae4-29e6545469ac","Type":"ContainerStarted","Data":"22c894186d62436439f63d72723e4639cb51088d3a42692a6d04c129bc7044b6"} Nov 03 02:18:01.588223 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:18:01.588172 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-898926/noten-model-898926-kserve-5b6b95d9bf-glsl2" event={"ID":"42963aa8-67e0-49e3-9ae4-29e6545469ac","Type":"ContainerStarted","Data":"5fa48d231e7f06c2d848f372cc88a8ea7736e9032e480bc9ec68e49eda202218"} Nov 03 02:18:01.589799 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:18:01.589771 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d8cc55-6f4d7f5cf4-njhfg" event={"ID":"8721e215-270a-4092-b0ef-16fb74a4b863","Type":"ContainerStarted","Data":"a92afd386cf5ba83a8293f695292b3745830db6278b12fc1f8377b2e4e1e1ee5"} Nov 03 02:18:01.590114 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:18:01.589857 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d8cc55-6f4d7f5cf4-njhfg" Nov 03 02:18:01.678322 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:18:01.678287 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-898926/noten-model-898926-kserve-5b6b95d9bf-glsl2" Nov 03 02:18:01.710906 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:18:01.710847 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-noten-898926/noten-model-898926-kserve-5b6b95d9bf-glsl2" podStartSLOduration=2.710832537 podStartE2EDuration="2.710832537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:17:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:18:01.708068202 +0000 UTC m=+3563.737550494" watchObservedRunningTime="2026-11-03 02:18:01.710832537 +0000 UTC m=+3563.740314831" Nov 03 02:18:01.829445 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:18:01.829392 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d8cc55-6f4d7f5cf4-njhfg" podStartSLOduration=1.976804572 podStartE2EDuration="2.829350316s" totalImagesPullingTime="852.545744ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:17:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:18:01.828729742 +0000 UTC m=+3563.858212039" watchObservedRunningTime="2026-11-03 02:18:01.829350316 +0000 UTC m=+3563.858832610" Nov 03 02:18:04.334894 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:18:04.334856 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:18:07.601758 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:18:07.601723 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d8cc55-6f4d7f5cf4-njhfg" Nov 03 02:18:09.077522 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:18:09.077478 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:18:10.678830 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:18:10.678792 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-898926/noten-model-898926-kserve-5b6b95d9bf-glsl2" Nov 03 02:18:10.691399 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:18:10.691356 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-898926/noten-model-898926-kserve-5b6b95d9bf-glsl2" Nov 03 02:18:11.664251 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:18:11.664220 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-898926/noten-model-898926-kserve-5b6b95d9bf-glsl2" Nov 03 02:18:12.091425 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:18:12.091392 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-898926/noten-model-898926-kserve-5b6b95d9bf-glsl2"] Nov 03 02:18:12.127684 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:18:12.127634 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:18:12.384428 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:18:12.384317 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:18:12.699066 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:18:12.698980 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:18:12.799703 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:18:12.799664 5437 secret.go:189] Couldn't get secret ai-tenant-e2e-noten-898926/noten-model-898926-kserve-self-signed-certs: secret "noten-model-898926-kserve-self-signed-certs" not found Nov 03 02:18:12.799918 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:18:12.799735 5437 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42963aa8-67e0-49e3-9ae4-29e6545469ac-tls-certs podName:42963aa8-67e0-49e3-9ae4-29e6545469ac nodeName:}" failed. No retries permitted until 2026-11-03 02:18:13.299719406 +0000 UTC m=+3575.329201694 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/42963aa8-67e0-49e3-9ae4-29e6545469ac-tls-certs") pod "noten-model-898926-kserve-5b6b95d9bf-glsl2" (UID: "42963aa8-67e0-49e3-9ae4-29e6545469ac") : secret "noten-model-898926-kserve-self-signed-certs" not found Nov 03 02:18:12.949839 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:18:12.949758 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:18:13.306411 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:18:13.306354 5437 secret.go:189] Couldn't get secret ai-tenant-e2e-noten-898926/noten-model-898926-kserve-self-signed-certs: secret "noten-model-898926-kserve-self-signed-certs" not found Nov 03 02:18:13.306811 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:18:13.306450 5437 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42963aa8-67e0-49e3-9ae4-29e6545469ac-tls-certs podName:42963aa8-67e0-49e3-9ae4-29e6545469ac nodeName:}" failed. No retries permitted until 2026-11-03 02:18:14.306436288 +0000 UTC m=+3576.335918561 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/42963aa8-67e0-49e3-9ae4-29e6545469ac-tls-certs") pod "noten-model-898926-kserve-5b6b95d9bf-glsl2" (UID: "42963aa8-67e0-49e3-9ae4-29e6545469ac") : secret "noten-model-898926-kserve-self-signed-certs" not found Nov 03 02:18:13.810717 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:18:13.810673 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:18:14.033892 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:18:14.033854 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 02:18:14.317786 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:18:14.317753 5437 secret.go:189] Couldn't get secret ai-tenant-e2e-noten-898926/noten-model-898926-kserve-self-signed-certs: secret "noten-model-898926-kserve-self-signed-certs" not found Nov 03 02:18:14.318202 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:18:14.317827 5437 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42963aa8-67e0-49e3-9ae4-29e6545469ac-tls-certs podName:42963aa8-67e0-49e3-9ae4-29e6545469ac nodeName:}" failed. No retries permitted until 2026-11-03 02:18:16.317814275 +0000 UTC m=+3578.347296552 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/42963aa8-67e0-49e3-9ae4-29e6545469ac-tls-certs") pod "noten-model-898926-kserve-5b6b95d9bf-glsl2" (UID: "42963aa8-67e0-49e3-9ae4-29e6545469ac") : secret "noten-model-898926-kserve-self-signed-certs" not found Nov 03 02:18:16.343441 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:18:16.343409 5437 secret.go:189] Couldn't get secret ai-tenant-e2e-noten-898926/noten-model-898926-kserve-self-signed-certs: secret "noten-model-898926-kserve-self-signed-certs" not found Nov 03 02:18:16.343806 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:18:16.343474 5437 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42963aa8-67e0-49e3-9ae4-29e6545469ac-tls-certs podName:42963aa8-67e0-49e3-9ae4-29e6545469ac nodeName:}" failed. No retries permitted until 2026-11-03 02:18:20.343460963 +0000 UTC m=+3582.372943238 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/42963aa8-67e0-49e3-9ae4-29e6545469ac-tls-certs") pod "noten-model-898926-kserve-5b6b95d9bf-glsl2" (UID: "42963aa8-67e0-49e3-9ae4-29e6545469ac") : secret "noten-model-898926-kserve-self-signed-certs" not found Nov 03 02:18:17.882399 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:18:17.882350 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:18:18.595258 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:18:18.595221 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:18:20.394199 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:18:20.394166 5437 secret.go:189] Couldn't get secret ai-tenant-e2e-noten-898926/noten-model-898926-kserve-self-signed-certs: secret "noten-model-898926-kserve-self-signed-certs" not found Nov 03 02:18:20.394607 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:18:20.394254 5437 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42963aa8-67e0-49e3-9ae4-29e6545469ac-tls-certs podName:42963aa8-67e0-49e3-9ae4-29e6545469ac nodeName:}" failed. No retries permitted until 2026-11-03 02:18:28.39423718 +0000 UTC m=+3590.423719459 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/42963aa8-67e0-49e3-9ae4-29e6545469ac-tls-certs") pod "noten-model-898926-kserve-5b6b95d9bf-glsl2" (UID: "42963aa8-67e0-49e3-9ae4-29e6545469ac") : secret "noten-model-898926-kserve-self-signed-certs" not found Nov 03 02:18:23.474327 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:18:23.474287 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:18:28.274713 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:18:28.274670 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:18:28.421439 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:18:28.421405 5437 secret.go:189] Couldn't get secret ai-tenant-e2e-noten-898926/noten-model-898926-kserve-self-signed-certs: secret "noten-model-898926-kserve-self-signed-certs" not found Nov 03 02:18:28.421632 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:18:28.421486 5437 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42963aa8-67e0-49e3-9ae4-29e6545469ac-tls-certs podName:42963aa8-67e0-49e3-9ae4-29e6545469ac nodeName:}" failed. No retries permitted until 2026-11-03 02:18:44.421471275 +0000 UTC m=+3606.450953548 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/42963aa8-67e0-49e3-9ae4-29e6545469ac-tls-certs") pod "noten-model-898926-kserve-5b6b95d9bf-glsl2" (UID: "42963aa8-67e0-49e3-9ae4-29e6545469ac") : secret "noten-model-898926-kserve-self-signed-certs" not found Nov 03 02:18:28.688294 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:18:28.688255 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-noten-898926/noten-model-898926-kserve-5b6b95d9bf-glsl2" podUID="42963aa8-67e0-49e3-9ae4-29e6545469ac" containerName="main" containerID="cri-o://22c894186d62436439f63d72723e4639cb51088d3a42692a6d04c129bc7044b6" gracePeriod=45 Nov 03 02:18:28.979749 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:18:28.979716 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-898926/noten-model-898926-kserve-5b6b95d9bf-glsl2" Nov 03 02:18:29.027649 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:18:29.027613 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42963aa8-67e0-49e3-9ae4-29e6545469ac-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42963aa8-67e0-49e3-9ae4-29e6545469ac-model-cache\") pod \"42963aa8-67e0-49e3-9ae4-29e6545469ac\" (UID: \"42963aa8-67e0-49e3-9ae4-29e6545469ac\") " Nov 03 02:18:29.027819 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:18:29.027676 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42963aa8-67e0-49e3-9ae4-29e6545469ac-kube-api-access-kvl8d\" (UniqueName: \"kubernetes.io/projected/42963aa8-67e0-49e3-9ae4-29e6545469ac-kube-api-access-kvl8d\") pod \"42963aa8-67e0-49e3-9ae4-29e6545469ac\" (UID: \"42963aa8-67e0-49e3-9ae4-29e6545469ac\") " Nov 03 02:18:29.027819 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:18:29.027721 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42963aa8-67e0-49e3-9ae4-29e6545469ac-home\" (UniqueName: \"kubernetes.io/empty-dir/42963aa8-67e0-49e3-9ae4-29e6545469ac-home\") pod \"42963aa8-67e0-49e3-9ae4-29e6545469ac\" (UID: \"42963aa8-67e0-49e3-9ae4-29e6545469ac\") " Nov 03 02:18:29.027819 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:18:29.027787 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42963aa8-67e0-49e3-9ae4-29e6545469ac-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42963aa8-67e0-49e3-9ae4-29e6545469ac-tmp-dir\") pod \"42963aa8-67e0-49e3-9ae4-29e6545469ac\" (UID: \"42963aa8-67e0-49e3-9ae4-29e6545469ac\") " Nov 03 02:18:29.027975 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:18:29.027957 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42963aa8-67e0-49e3-9ae4-29e6545469ac-dshm\" (UniqueName: \"kubernetes.io/empty-dir/42963aa8-67e0-49e3-9ae4-29e6545469ac-dshm\") pod \"42963aa8-67e0-49e3-9ae4-29e6545469ac\" (UID: \"42963aa8-67e0-49e3-9ae4-29e6545469ac\") " Nov 03 02:18:29.028024 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:18:29.027992 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42963aa8-67e0-49e3-9ae4-29e6545469ac-model-cache" pod "42963aa8-67e0-49e3-9ae4-29e6545469ac" (UID: "42963aa8-67e0-49e3-9ae4-29e6545469ac"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:18:29.028091 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:18:29.028039 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/42963aa8-67e0-49e3-9ae4-29e6545469ac-tls-certs\" (UniqueName: \"kubernetes.io/secret/42963aa8-67e0-49e3-9ae4-29e6545469ac-tls-certs\") pod \"42963aa8-67e0-49e3-9ae4-29e6545469ac\" (UID: \"42963aa8-67e0-49e3-9ae4-29e6545469ac\") " Nov 03 02:18:29.028091 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:18:29.028061 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42963aa8-67e0-49e3-9ae4-29e6545469ac-home" pod "42963aa8-67e0-49e3-9ae4-29e6545469ac" (UID: "42963aa8-67e0-49e3-9ae4-29e6545469ac"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:18:29.028204 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:18:29.028106 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42963aa8-67e0-49e3-9ae4-29e6545469ac-tmp-dir" pod "42963aa8-67e0-49e3-9ae4-29e6545469ac" (UID: "42963aa8-67e0-49e3-9ae4-29e6545469ac"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:18:29.028530 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:18:29.028505 5437 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42963aa8-67e0-49e3-9ae4-29e6545469ac-model-cache\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:18:29.028638 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:18:29.028530 5437 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42963aa8-67e0-49e3-9ae4-29e6545469ac-home\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:18:29.028638 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:18:29.028545 5437 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42963aa8-67e0-49e3-9ae4-29e6545469ac-tmp-dir\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:18:29.030236 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:18:29.030210 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42963aa8-67e0-49e3-9ae4-29e6545469ac-kube-api-access-kvl8d" pod "42963aa8-67e0-49e3-9ae4-29e6545469ac" (UID: "42963aa8-67e0-49e3-9ae4-29e6545469ac"). InnerVolumeSpecName "kube-api-access-kvl8d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:18:29.030506 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:18:29.030488 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42963aa8-67e0-49e3-9ae4-29e6545469ac-dshm" pod "42963aa8-67e0-49e3-9ae4-29e6545469ac" (UID: "42963aa8-67e0-49e3-9ae4-29e6545469ac"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:18:29.030574 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:18:29.030511 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42963aa8-67e0-49e3-9ae4-29e6545469ac-tls-certs" pod "42963aa8-67e0-49e3-9ae4-29e6545469ac" (UID: "42963aa8-67e0-49e3-9ae4-29e6545469ac"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:18:29.130165 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:18:29.130126 5437 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42963aa8-67e0-49e3-9ae4-29e6545469ac-dshm\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:18:29.130165 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:18:29.130164 5437 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42963aa8-67e0-49e3-9ae4-29e6545469ac-tls-certs\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:18:29.130489 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:18:29.130181 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvl8d\" (UniqueName: \"kubernetes.io/projected/42963aa8-67e0-49e3-9ae4-29e6545469ac-kube-api-access-kvl8d\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:18:29.807385 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:18:29.807337 5437 generic.go:356] "Generic (PLEG): container finished" podID="6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea" containerID="659c9112015d4f3ab0715fd52664d159b8a66d7dc8ba8c91c0ff6a85772096b4" exitCode=0 Nov 03 02:18:29.807875 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:18:29.807414 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-50ca6d-openshift-default-6b7b98d8c7-499js" event={"ID":"6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea","Type":"ContainerDied","Data":"659c9112015d4f3ab0715fd52664d159b8a66d7dc8ba8c91c0ff6a85772096b4"} Nov 03 02:18:29.807956 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:18:29.807891 5437 scope.go:122] "RemoveContainer" containerID="659c9112015d4f3ab0715fd52664d159b8a66d7dc8ba8c91c0ff6a85772096b4" Nov 03 02:18:29.809117 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:18:29.809087 5437 generic.go:356] "Generic (PLEG): container finished" podID="42963aa8-67e0-49e3-9ae4-29e6545469ac" containerID="22c894186d62436439f63d72723e4639cb51088d3a42692a6d04c129bc7044b6" exitCode=0 Nov 03 02:18:29.809218 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:18:29.809129 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-898926/noten-model-898926-kserve-5b6b95d9bf-glsl2" Nov 03 02:18:29.809283 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:18:29.809131 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-898926/noten-model-898926-kserve-5b6b95d9bf-glsl2" event={"ID":"42963aa8-67e0-49e3-9ae4-29e6545469ac","Type":"ContainerDied","Data":"22c894186d62436439f63d72723e4639cb51088d3a42692a6d04c129bc7044b6"} Nov 03 02:18:29.809283 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:18:29.809264 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-898926/noten-model-898926-kserve-5b6b95d9bf-glsl2" event={"ID":"42963aa8-67e0-49e3-9ae4-29e6545469ac","Type":"ContainerDied","Data":"5fa48d231e7f06c2d848f372cc88a8ea7736e9032e480bc9ec68e49eda202218"} Nov 03 02:18:29.809433 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:18:29.809293 5437 scope.go:122] "RemoveContainer" containerID="22c894186d62436439f63d72723e4639cb51088d3a42692a6d04c129bc7044b6" Nov 03 02:18:29.809791 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:18:29.809756 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 02:18:29.809893 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:18:29.809837 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 02:18:29.809893 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:18:29.809875 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 02:18:29.825003 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:18:29.824985 5437 scope.go:122] "RemoveContainer" containerID="22c894186d62436439f63d72723e4639cb51088d3a42692a6d04c129bc7044b6" Nov 03 02:18:29.825345 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:18:29.825320 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22c894186d62436439f63d72723e4639cb51088d3a42692a6d04c129bc7044b6\": container with ID starting with 22c894186d62436439f63d72723e4639cb51088d3a42692a6d04c129bc7044b6 not found: ID does not exist" containerID="22c894186d62436439f63d72723e4639cb51088d3a42692a6d04c129bc7044b6" Nov 03 02:18:29.825449 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:18:29.825358 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22c894186d62436439f63d72723e4639cb51088d3a42692a6d04c129bc7044b6"} err="failed to get container status \"22c894186d62436439f63d72723e4639cb51088d3a42692a6d04c129bc7044b6\": rpc error: code = NotFound desc = could not find container \"22c894186d62436439f63d72723e4639cb51088d3a42692a6d04c129bc7044b6\": container with ID starting with 22c894186d62436439f63d72723e4639cb51088d3a42692a6d04c129bc7044b6 not found: ID does not exist" Nov 03 02:18:30.137731 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:18:30.137697 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-898926/noten-model-898926-kserve-5b6b95d9bf-glsl2"] Nov 03 02:18:30.142004 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:18:30.141980 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-898926/noten-model-898926-kserve-5b6b95d9bf-glsl2"] Nov 03 02:18:30.817104 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:18:30.817063 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-50ca6d-openshift-default-6b7b98d8c7-499js" event={"ID":"6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea","Type":"ContainerStarted","Data":"69f899906e8c170453bf9a7a42d26165c8df0a3ac53a5a98b28acb4d21b7cd47"} Nov 03 02:18:31.656292 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:18:31.656253 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-50ca6d-openshift-default-6b7b98d8c7-499js" Nov 03 02:18:31.656507 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:18:31.656490 5437 patch_prober.go:29] interesting pod/e2e-autores-50ca6d-openshift-default-6b7b98d8c7-499js container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body= Nov 03 02:18:31.656560 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:18:31.656537 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-50ca6d-openshift-default-6b7b98d8c7-499js" podUID="6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" Nov 03 02:18:32.089165 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:18:32.089131 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42963aa8-67e0-49e3-9ae4-29e6545469ac" path="/var/lib/kubelet/pods/42963aa8-67e0-49e3-9ae4-29e6545469ac/volumes" Nov 03 02:18:32.657018 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:18:32.656978 5437 patch_prober.go:29] interesting pod/e2e-autores-50ca6d-openshift-default-6b7b98d8c7-499js container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body= Nov 03 02:18:32.657203 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:18:32.657054 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-50ca6d-openshift-default-6b7b98d8c7-499js" podUID="6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" Nov 03 02:18:33.049926 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:18:33.049891 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:18:33.656849 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:18:33.656807 5437 patch_prober.go:29] interesting pod/e2e-autores-50ca6d-openshift-default-6b7b98d8c7-499js container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body= Nov 03 02:18:33.657334 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:18:33.656901 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-50ca6d-openshift-default-6b7b98d8c7-499js" podUID="6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" Nov 03 02:18:34.657039 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:18:34.656996 5437 patch_prober.go:29] interesting pod/e2e-autores-50ca6d-openshift-default-6b7b98d8c7-499js container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body= Nov 03 02:18:34.657540 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:18:34.657058 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-50ca6d-openshift-default-6b7b98d8c7-499js" podUID="6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" Nov 03 02:18:35.656610 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:18:35.656566 5437 patch_prober.go:29] interesting pod/e2e-autores-50ca6d-openshift-default-6b7b98d8c7-499js container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body= Nov 03 02:18:35.656800 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:18:35.656634 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-50ca6d-openshift-default-6b7b98d8c7-499js" podUID="6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" Nov 03 02:18:36.661226 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:18:36.661189 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-50ca6d-openshift-default-6b7b98d8c7-499js" Nov 03 02:18:36.661742 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:18:36.661608 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-50ca6d-openshift-default-6b7b98d8c7-499js" Nov 03 02:18:36.662338 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:18:36.662320 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-50ca6d-openshift-default-6b7b98d8c7-499js" Nov 03 02:18:37.962437 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:18:37.962398 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:18:42.771688 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:18:42.771644 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:18:44.035132 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:18:44.035099 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 02:18:47.499127 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:18:47.499038 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:18:52.221333 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:18:52.221290 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:18:54.924873 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:18:54.924835 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:18:56.333031 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:18:56.332988 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:18:56.537645 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:18:56.537611 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:18:56.924612 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:18:56.924582 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:18:57.148547 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:18:57.148443 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:18:57.351113 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:18:57.351075 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:19:00.806738 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:00.806693 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:19:01.089270 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:01.089184 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:19:01.298200 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:01.298160 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:19:01.508404 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:01.508305 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:19:02.098087 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:02.098050 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:19:02.370863 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:02.370770 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:19:02.728005 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:02.727905 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:19:02.966281 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:02.966234 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:19:06.576154 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:06.576113 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:19:06.778098 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:06.778061 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:19:06.921485 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:06.921392 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-bd872a-7b5f4b9979-g4cqr"] Nov 03 02:19:06.922113 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:06.922095 5437 cpu_manager.go:414] "containerMap: removing container" podUID="42963aa8-67e0-49e3-9ae4-29e6545469ac" containerName="main" Nov 03 02:19:06.922113 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:06.922113 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42963aa8-67e0-49e3-9ae4-29e6545469ac" containerName="main" Nov 03 02:19:06.922279 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:06.922212 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="42963aa8-67e0-49e3-9ae4-29e6545469ac" containerName="main" Nov 03 02:19:06.924821 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:06.924797 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-bd872a-7b5f4b9979-g4cqr" Nov 03 02:19:06.979709 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:06.979670 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-bd872a-7b5f4b9979-g4cqr"] Nov 03 02:19:06.999506 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:06.999462 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32db9a05-1703-4fbc-b999-fdba1981f184-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-bd872a-7b5f4b9979-g4cqr\" (UID: \"32db9a05-1703-4fbc-b999-fdba1981f184\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-bd872a-7b5f4b9979-g4cqr" Nov 03 02:19:06.999988 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:06.999952 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hhgp\" (UniqueName: \"kubernetes.io/projected/32db9a05-1703-4fbc-b999-fdba1981f184-kube-api-access-4hhgp\") pod \"payload-pre-processing-e2e-extmod-a-bd872a-7b5f4b9979-g4cqr\" (UID: \"32db9a05-1703-4fbc-b999-fdba1981f184\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-bd872a-7b5f4b9979-g4cqr" Nov 03 02:19:07.027574 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:07.027535 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-bd872a\"" Nov 03 02:19:07.027754 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:07.027717 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-bd872a-dockercfg-pq7wt\"" Nov 03 02:19:07.101252 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:07.101217 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hhgp\" (UniqueName: \"kubernetes.io/projected/32db9a05-1703-4fbc-b999-fdba1981f184-kube-api-access-4hhgp\") pod \"payload-pre-processing-e2e-extmod-a-bd872a-7b5f4b9979-g4cqr\" (UID: \"32db9a05-1703-4fbc-b999-fdba1981f184\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-bd872a-7b5f4b9979-g4cqr" Nov 03 02:19:07.101447 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:07.101282 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32db9a05-1703-4fbc-b999-fdba1981f184-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-bd872a-7b5f4b9979-g4cqr\" (UID: \"32db9a05-1703-4fbc-b999-fdba1981f184\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-bd872a-7b5f4b9979-g4cqr" Nov 03 02:19:07.102064 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:07.102042 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32db9a05-1703-4fbc-b999-fdba1981f184-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-bd872a-7b5f4b9979-g4cqr\" (UID: \"32db9a05-1703-4fbc-b999-fdba1981f184\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-bd872a-7b5f4b9979-g4cqr" Nov 03 02:19:07.164328 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:07.164298 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hhgp\" (UniqueName: \"kubernetes.io/projected/32db9a05-1703-4fbc-b999-fdba1981f184-kube-api-access-4hhgp\") pod \"payload-pre-processing-e2e-extmod-a-bd872a-7b5f4b9979-g4cqr\" (UID: \"32db9a05-1703-4fbc-b999-fdba1981f184\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-bd872a-7b5f4b9979-g4cqr" Nov 03 02:19:07.240204 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:07.240118 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-bd872a-7b5f4b9979-g4cqr" Nov 03 02:19:07.618920 ip-10-0-136-16 kubenswrapper[5437]: W1103 02:19:07.614162 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32db9a05_1703_4fbc_b999_fdba1981f184.slice/crio-5a7bdc287915f3ab024cb6ce81c6cfd0c51b1f7ec22d796cf22b9744e7469f54 WatchSource:0}: Error finding container 5a7bdc287915f3ab024cb6ce81c6cfd0c51b1f7ec22d796cf22b9744e7469f54: Status 404 returned error can't find the container with id 5a7bdc287915f3ab024cb6ce81c6cfd0c51b1f7ec22d796cf22b9744e7469f54 Nov 03 02:19:07.618920 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:07.617191 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-bd872a-7b5f4b9979-g4cqr"] Nov 03 02:19:07.624271 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:07.624243 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:19:08.022745 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:08.022646 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:19:08.090011 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:08.089977 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-bd872a-7b5f4b9979-g4cqr" event={"ID":"32db9a05-1703-4fbc-b999-fdba1981f184","Type":"ContainerStarted","Data":"1f8ff2b0014afd456823ac4d9b24d45cecdca1b3c11b2ff2fcf612af6ecf5f6b"} Nov 03 02:19:08.090178 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:08.090032 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-bd872a-7b5f4b9979-g4cqr" Nov 03 02:19:08.090178 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:08.090049 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-bd872a-7b5f4b9979-g4cqr" event={"ID":"32db9a05-1703-4fbc-b999-fdba1981f184","Type":"ContainerStarted","Data":"5a7bdc287915f3ab024cb6ce81c6cfd0c51b1f7ec22d796cf22b9744e7469f54"} Nov 03 02:19:08.209736 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:08.209687 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-bd872a-7b5f4b9979-g4cqr" podStartSLOduration=2.209674355 podStartE2EDuration="2.209674355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:19:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:19:08.208442311 +0000 UTC m=+3630.237924606" watchObservedRunningTime="2026-11-03 02:19:08.209674355 +0000 UTC m=+3630.239156650" Nov 03 02:19:08.224601 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:08.224567 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:19:11.168737 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:11.168695 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:19:11.376277 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:11.376236 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:19:11.685288 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:11.685254 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-3fb07e-7678cbbc5-kpm8s"] Nov 03 02:19:11.689593 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:11.689572 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-3fb07e-7678cbbc5-kpm8s" Nov 03 02:19:11.745175 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:11.745137 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-3fb07e-7678cbbc5-kpm8s"] Nov 03 02:19:11.792339 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:11.792301 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-3fb07e-dockercfg-jjnzk\"" Nov 03 02:19:11.792339 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:11.792301 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-3fb07e\"" Nov 03 02:19:11.864163 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:11.864124 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lmnf\" (UniqueName: \"kubernetes.io/projected/db26a4c8-466a-4c1e-ba51-30bd6a076791-kube-api-access-8lmnf\") pod \"payload-processing-e2e-extmod-b-3fb07e-7678cbbc5-kpm8s\" (UID: \"db26a4c8-466a-4c1e-ba51-30bd6a076791\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-3fb07e-7678cbbc5-kpm8s" Nov 03 02:19:11.864326 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:11.864277 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db26a4c8-466a-4c1e-ba51-30bd6a076791-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-3fb07e-7678cbbc5-kpm8s\" (UID: \"db26a4c8-466a-4c1e-ba51-30bd6a076791\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-3fb07e-7678cbbc5-kpm8s" Nov 03 02:19:11.965837 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:11.965750 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lmnf\" (UniqueName: \"kubernetes.io/projected/db26a4c8-466a-4c1e-ba51-30bd6a076791-kube-api-access-8lmnf\") pod \"payload-processing-e2e-extmod-b-3fb07e-7678cbbc5-kpm8s\" (UID: \"db26a4c8-466a-4c1e-ba51-30bd6a076791\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-3fb07e-7678cbbc5-kpm8s" Nov 03 02:19:11.965998 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:11.965880 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db26a4c8-466a-4c1e-ba51-30bd6a076791-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-3fb07e-7678cbbc5-kpm8s\" (UID: \"db26a4c8-466a-4c1e-ba51-30bd6a076791\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-3fb07e-7678cbbc5-kpm8s" Nov 03 02:19:11.966666 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:11.966644 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db26a4c8-466a-4c1e-ba51-30bd6a076791-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-3fb07e-7678cbbc5-kpm8s\" (UID: \"db26a4c8-466a-4c1e-ba51-30bd6a076791\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-3fb07e-7678cbbc5-kpm8s" Nov 03 02:19:12.028697 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:12.028668 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lmnf\" (UniqueName: \"kubernetes.io/projected/db26a4c8-466a-4c1e-ba51-30bd6a076791-kube-api-access-8lmnf\") pod \"payload-processing-e2e-extmod-b-3fb07e-7678cbbc5-kpm8s\" (UID: \"db26a4c8-466a-4c1e-ba51-30bd6a076791\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-3fb07e-7678cbbc5-kpm8s" Nov 03 02:19:12.303400 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:12.303358 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-3fb07e-7678cbbc5-kpm8s" Nov 03 02:19:12.673487 ip-10-0-136-16 kubenswrapper[5437]: W1103 02:19:12.673452 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb26a4c8_466a_4c1e_ba51_30bd6a076791.slice/crio-1939c741cd700ca8982ab83ded08319e45eabe8edd54ef32f8b99068e87bc921 WatchSource:0}: Error finding container 1939c741cd700ca8982ab83ded08319e45eabe8edd54ef32f8b99068e87bc921: Status 404 returned error can't find the container with id 1939c741cd700ca8982ab83ded08319e45eabe8edd54ef32f8b99068e87bc921 Nov 03 02:19:12.676629 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:12.676605 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-3fb07e-7678cbbc5-kpm8s"] Nov 03 02:19:12.872975 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:12.872934 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:19:13.127210 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:13.127170 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-3fb07e-7678cbbc5-kpm8s" event={"ID":"db26a4c8-466a-4c1e-ba51-30bd6a076791","Type":"ContainerStarted","Data":"b4c5a526f4da2db04711b6287428a2d9812139314c8f8f20342887f00fc1bba9"} Nov 03 02:19:13.127443 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:13.127239 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-3fb07e-7678cbbc5-kpm8s" event={"ID":"db26a4c8-466a-4c1e-ba51-30bd6a076791","Type":"ContainerStarted","Data":"1939c741cd700ca8982ab83ded08319e45eabe8edd54ef32f8b99068e87bc921"} Nov 03 02:19:13.127443 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:13.127271 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-3fb07e-7678cbbc5-kpm8s" Nov 03 02:19:13.249709 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:13.249645 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-3fb07e-7678cbbc5-kpm8s" podStartSLOduration=2.24962655 podStartE2EDuration="2.24962655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:19:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:19:13.245770914 +0000 UTC m=+3635.275253208" watchObservedRunningTime="2026-11-03 02:19:13.24962655 +0000 UTC m=+3635.279108854" Nov 03 02:19:14.035889 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:19:14.035853 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 02:19:17.928706 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:17.928669 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:19:19.098923 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:19.098894 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-bd872a-7b5f4b9979-g4cqr" Nov 03 02:19:22.977629 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:22.977579 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:19:23.334795 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:23.334757 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:19:24.136709 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:24.136680 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-3fb07e-7678cbbc5-kpm8s" Nov 03 02:19:24.798290 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:24.798235 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-tz9hh"] Nov 03 02:19:24.802963 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:24.802938 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-tz9hh" Nov 03 02:19:24.856892 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:24.856851 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-4f87d1-openshift-default-dockercfg-mk9lx\"" Nov 03 02:19:24.911505 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:24.911462 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-tz9hh"] Nov 03 02:19:24.926465 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:24.926436 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5837de8-ead2-4681-8689-3d5fb3bd3ae1-istio-token\") pod \"e2e-extmod-b-4f87d1-openshift-default-7cb54948df-tz9hh\" (UID: \"a5837de8-ead2-4681-8689-3d5fb3bd3ae1\") " pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-tz9hh" Nov 03 02:19:24.926619 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:24.926506 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqcmp\" (UniqueName: \"kubernetes.io/projected/a5837de8-ead2-4681-8689-3d5fb3bd3ae1-kube-api-access-hqcmp\") pod \"e2e-extmod-b-4f87d1-openshift-default-7cb54948df-tz9hh\" (UID: \"a5837de8-ead2-4681-8689-3d5fb3bd3ae1\") " pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-tz9hh" Nov 03 02:19:24.926673 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:24.926646 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5837de8-ead2-4681-8689-3d5fb3bd3ae1-credential-socket\") pod \"e2e-extmod-b-4f87d1-openshift-default-7cb54948df-tz9hh\" (UID: \"a5837de8-ead2-4681-8689-3d5fb3bd3ae1\") " pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-tz9hh" Nov 03 02:19:24.926712 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:24.926692 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5837de8-ead2-4681-8689-3d5fb3bd3ae1-istio-envoy\") pod \"e2e-extmod-b-4f87d1-openshift-default-7cb54948df-tz9hh\" (UID: \"a5837de8-ead2-4681-8689-3d5fb3bd3ae1\") " pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-tz9hh" Nov 03 02:19:24.926829 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:24.926814 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5837de8-ead2-4681-8689-3d5fb3bd3ae1-istio-podinfo\") pod \"e2e-extmod-b-4f87d1-openshift-default-7cb54948df-tz9hh\" (UID: \"a5837de8-ead2-4681-8689-3d5fb3bd3ae1\") " pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-tz9hh" Nov 03 02:19:24.926895 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:24.926880 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5837de8-ead2-4681-8689-3d5fb3bd3ae1-workload-socket\") pod \"e2e-extmod-b-4f87d1-openshift-default-7cb54948df-tz9hh\" (UID: \"a5837de8-ead2-4681-8689-3d5fb3bd3ae1\") " pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-tz9hh" Nov 03 02:19:24.926933 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:24.926913 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5837de8-ead2-4681-8689-3d5fb3bd3ae1-istiod-ca-cert\") pod \"e2e-extmod-b-4f87d1-openshift-default-7cb54948df-tz9hh\" (UID: \"a5837de8-ead2-4681-8689-3d5fb3bd3ae1\") " pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-tz9hh" Nov 03 02:19:24.926968 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:24.926939 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5837de8-ead2-4681-8689-3d5fb3bd3ae1-istio-data\") pod \"e2e-extmod-b-4f87d1-openshift-default-7cb54948df-tz9hh\" (UID: \"a5837de8-ead2-4681-8689-3d5fb3bd3ae1\") " pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-tz9hh" Nov 03 02:19:24.927048 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:24.927035 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5837de8-ead2-4681-8689-3d5fb3bd3ae1-workload-certs\") pod \"e2e-extmod-b-4f87d1-openshift-default-7cb54948df-tz9hh\" (UID: \"a5837de8-ead2-4681-8689-3d5fb3bd3ae1\") " pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-tz9hh" Nov 03 02:19:25.027759 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:25.027721 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5837de8-ead2-4681-8689-3d5fb3bd3ae1-istio-data\") pod \"e2e-extmod-b-4f87d1-openshift-default-7cb54948df-tz9hh\" (UID: \"a5837de8-ead2-4681-8689-3d5fb3bd3ae1\") " pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-tz9hh" Nov 03 02:19:25.027759 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:25.027762 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5837de8-ead2-4681-8689-3d5fb3bd3ae1-workload-certs\") pod \"e2e-extmod-b-4f87d1-openshift-default-7cb54948df-tz9hh\" (UID: \"a5837de8-ead2-4681-8689-3d5fb3bd3ae1\") " pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-tz9hh" Nov 03 02:19:25.027992 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:25.027801 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5837de8-ead2-4681-8689-3d5fb3bd3ae1-istio-token\") pod \"e2e-extmod-b-4f87d1-openshift-default-7cb54948df-tz9hh\" (UID: \"a5837de8-ead2-4681-8689-3d5fb3bd3ae1\") " pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-tz9hh" Nov 03 02:19:25.027992 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:25.027851 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqcmp\" (UniqueName: \"kubernetes.io/projected/a5837de8-ead2-4681-8689-3d5fb3bd3ae1-kube-api-access-hqcmp\") pod \"e2e-extmod-b-4f87d1-openshift-default-7cb54948df-tz9hh\" (UID: \"a5837de8-ead2-4681-8689-3d5fb3bd3ae1\") " pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-tz9hh" Nov 03 02:19:25.027992 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:25.027910 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5837de8-ead2-4681-8689-3d5fb3bd3ae1-credential-socket\") pod \"e2e-extmod-b-4f87d1-openshift-default-7cb54948df-tz9hh\" (UID: \"a5837de8-ead2-4681-8689-3d5fb3bd3ae1\") " pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-tz9hh" Nov 03 02:19:25.027992 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:25.027942 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5837de8-ead2-4681-8689-3d5fb3bd3ae1-istio-envoy\") pod \"e2e-extmod-b-4f87d1-openshift-default-7cb54948df-tz9hh\" (UID: \"a5837de8-ead2-4681-8689-3d5fb3bd3ae1\") " pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-tz9hh" Nov 03 02:19:25.028209 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:25.028017 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5837de8-ead2-4681-8689-3d5fb3bd3ae1-istio-podinfo\") pod \"e2e-extmod-b-4f87d1-openshift-default-7cb54948df-tz9hh\" (UID: \"a5837de8-ead2-4681-8689-3d5fb3bd3ae1\") " pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-tz9hh" Nov 03 02:19:25.028209 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:25.028098 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5837de8-ead2-4681-8689-3d5fb3bd3ae1-workload-socket\") pod \"e2e-extmod-b-4f87d1-openshift-default-7cb54948df-tz9hh\" (UID: \"a5837de8-ead2-4681-8689-3d5fb3bd3ae1\") " pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-tz9hh" Nov 03 02:19:25.028209 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:25.028139 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5837de8-ead2-4681-8689-3d5fb3bd3ae1-istiod-ca-cert\") pod \"e2e-extmod-b-4f87d1-openshift-default-7cb54948df-tz9hh\" (UID: \"a5837de8-ead2-4681-8689-3d5fb3bd3ae1\") " pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-tz9hh" Nov 03 02:19:25.028209 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:25.028158 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5837de8-ead2-4681-8689-3d5fb3bd3ae1-istio-data\") pod \"e2e-extmod-b-4f87d1-openshift-default-7cb54948df-tz9hh\" (UID: \"a5837de8-ead2-4681-8689-3d5fb3bd3ae1\") " pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-tz9hh" Nov 03 02:19:25.028455 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:25.028216 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5837de8-ead2-4681-8689-3d5fb3bd3ae1-workload-certs\") pod \"e2e-extmod-b-4f87d1-openshift-default-7cb54948df-tz9hh\" (UID: \"a5837de8-ead2-4681-8689-3d5fb3bd3ae1\") " pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-tz9hh" Nov 03 02:19:25.028455 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:25.028255 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5837de8-ead2-4681-8689-3d5fb3bd3ae1-credential-socket\") pod \"e2e-extmod-b-4f87d1-openshift-default-7cb54948df-tz9hh\" (UID: \"a5837de8-ead2-4681-8689-3d5fb3bd3ae1\") " pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-tz9hh" Nov 03 02:19:25.028566 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:25.028518 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5837de8-ead2-4681-8689-3d5fb3bd3ae1-workload-socket\") pod \"e2e-extmod-b-4f87d1-openshift-default-7cb54948df-tz9hh\" (UID: \"a5837de8-ead2-4681-8689-3d5fb3bd3ae1\") " pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-tz9hh" Nov 03 02:19:25.028830 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:25.028812 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5837de8-ead2-4681-8689-3d5fb3bd3ae1-istiod-ca-cert\") pod \"e2e-extmod-b-4f87d1-openshift-default-7cb54948df-tz9hh\" (UID: \"a5837de8-ead2-4681-8689-3d5fb3bd3ae1\") " pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-tz9hh" Nov 03 02:19:25.030489 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:25.030463 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5837de8-ead2-4681-8689-3d5fb3bd3ae1-istio-envoy\") pod \"e2e-extmod-b-4f87d1-openshift-default-7cb54948df-tz9hh\" (UID: \"a5837de8-ead2-4681-8689-3d5fb3bd3ae1\") " pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-tz9hh" Nov 03 02:19:25.030665 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:25.030646 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5837de8-ead2-4681-8689-3d5fb3bd3ae1-istio-podinfo\") pod \"e2e-extmod-b-4f87d1-openshift-default-7cb54948df-tz9hh\" (UID: \"a5837de8-ead2-4681-8689-3d5fb3bd3ae1\") " pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-tz9hh" Nov 03 02:19:25.091018 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:25.090942 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5837de8-ead2-4681-8689-3d5fb3bd3ae1-istio-token\") pod \"e2e-extmod-b-4f87d1-openshift-default-7cb54948df-tz9hh\" (UID: \"a5837de8-ead2-4681-8689-3d5fb3bd3ae1\") " pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-tz9hh" Nov 03 02:19:25.091158 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:25.091062 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqcmp\" (UniqueName: \"kubernetes.io/projected/a5837de8-ead2-4681-8689-3d5fb3bd3ae1-kube-api-access-hqcmp\") pod \"e2e-extmod-b-4f87d1-openshift-default-7cb54948df-tz9hh\" (UID: \"a5837de8-ead2-4681-8689-3d5fb3bd3ae1\") " pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-tz9hh" Nov 03 02:19:25.117060 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:25.117027 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-tz9hh" Nov 03 02:19:25.379704 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:25.379624 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:19:25.542558 ip-10-0-136-16 kubenswrapper[5437]: W1103 02:19:25.542523 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5837de8_ead2_4681_8689_3d5fb3bd3ae1.slice/crio-759783293a6763e80ac67a64a3d49bee642a2da852123ddba5326024c3416542 WatchSource:0}: Error finding container 759783293a6763e80ac67a64a3d49bee642a2da852123ddba5326024c3416542: Status 404 returned error can't find the container with id 759783293a6763e80ac67a64a3d49bee642a2da852123ddba5326024c3416542 Nov 03 02:19:25.546078 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:25.546050 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-tz9hh"] Nov 03 02:19:25.546297 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:25.546266 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 02:19:25.546379 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:25.546327 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 02:19:25.546379 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:25.546356 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 02:19:26.212201 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:26.212164 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-tz9hh" event={"ID":"a5837de8-ead2-4681-8689-3d5fb3bd3ae1","Type":"ContainerStarted","Data":"5af73d02f93d7bd464fd88c19f774b1b96de6159e16946af43b5994cf03b863c"} Nov 03 02:19:26.212397 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:26.212221 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-tz9hh" event={"ID":"a5837de8-ead2-4681-8689-3d5fb3bd3ae1","Type":"ContainerStarted","Data":"759783293a6763e80ac67a64a3d49bee642a2da852123ddba5326024c3416542"} Nov 03 02:19:26.443685 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:26.443614 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-tz9hh" podStartSLOduration=2.443591708 podStartE2EDuration="2.443591708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:19:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:19:26.389452638 +0000 UTC m=+3648.418934937" watchObservedRunningTime="2026-11-03 02:19:26.443591708 +0000 UTC m=+3648.473074007" Nov 03 02:19:27.141391 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:27.140745 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-tz9hh" Nov 03 02:19:27.144327 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:27.144297 5437 patch_prober.go:29] interesting pod/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-tz9hh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body= Nov 03 02:19:27.144490 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:27.144350 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-tz9hh" podUID="a5837de8-ead2-4681-8689-3d5fb3bd3ae1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" Nov 03 02:19:28.037410 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:28.031245 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:19:28.140390 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:28.137756 5437 patch_prober.go:29] interesting pod/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-tz9hh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body= Nov 03 02:19:28.140390 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:28.137823 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-tz9hh" podUID="a5837de8-ead2-4681-8689-3d5fb3bd3ae1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" Nov 03 02:19:29.140009 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:29.139976 5437 patch_prober.go:29] interesting pod/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-tz9hh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body= Nov 03 02:19:29.140678 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:29.140644 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-tz9hh" podUID="a5837de8-ead2-4681-8689-3d5fb3bd3ae1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" Nov 03 02:19:29.343963 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:29.343923 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:19:30.142389 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:30.138507 5437 patch_prober.go:29] interesting pod/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-tz9hh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body= Nov 03 02:19:30.142389 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:30.138575 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-tz9hh" podUID="a5837de8-ead2-4681-8689-3d5fb3bd3ae1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" Nov 03 02:19:31.117306 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:31.117272 5437 patch_prober.go:29] interesting pod/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-tz9hh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body= Nov 03 02:19:31.117539 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:31.117334 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-tz9hh" podUID="a5837de8-ead2-4681-8689-3d5fb3bd3ae1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" Nov 03 02:19:31.315558 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:31.315518 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-50ca6d-774c6bf95-b26nc"] Nov 03 02:19:31.756015 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:31.755981 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:19:31.978502 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:31.978455 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:19:32.123399 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:32.123355 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-tz9hh" Nov 03 02:19:32.123638 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:32.123617 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-tz9hh" Nov 03 02:19:32.124270 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:32.124252 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-tz9hh" Nov 03 02:19:33.500552 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:33.500514 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:19:35.210595 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:35.210550 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:19:35.468219 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:35.468134 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:19:36.316502 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:36.316444 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-50ca6d-774c6bf95-b26nc" podUID="50d03554-8ebc-498c-848f-0f0dbc24dd41" containerName="payload-processing" containerID="cri-o://0183c1b9d6dbe402818660d32b905f2435403220dc6a2919210619070e3592d9" gracePeriod=25 Nov 03 02:19:36.592798 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:36.592769 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-50ca6d-774c6bf95-b26nc" Nov 03 02:19:36.677777 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:36.677742 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50d03554-8ebc-498c-848f-0f0dbc24dd41-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50d03554-8ebc-498c-848f-0f0dbc24dd41-plugins-config-volume\") pod \"50d03554-8ebc-498c-848f-0f0dbc24dd41\" (UID: \"50d03554-8ebc-498c-848f-0f0dbc24dd41\") " Nov 03 02:19:36.677956 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:36.677884 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50d03554-8ebc-498c-848f-0f0dbc24dd41-kube-api-access-bgqm6\" (UniqueName: \"kubernetes.io/projected/50d03554-8ebc-498c-848f-0f0dbc24dd41-kube-api-access-bgqm6\") pod \"50d03554-8ebc-498c-848f-0f0dbc24dd41\" (UID: \"50d03554-8ebc-498c-848f-0f0dbc24dd41\") " Nov 03 02:19:36.678186 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:36.678160 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50d03554-8ebc-498c-848f-0f0dbc24dd41-plugins-config-volume" pod "50d03554-8ebc-498c-848f-0f0dbc24dd41" (UID: "50d03554-8ebc-498c-848f-0f0dbc24dd41"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:19:36.679968 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:36.679948 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50d03554-8ebc-498c-848f-0f0dbc24dd41-kube-api-access-bgqm6" pod "50d03554-8ebc-498c-848f-0f0dbc24dd41" (UID: "50d03554-8ebc-498c-848f-0f0dbc24dd41"). InnerVolumeSpecName "kube-api-access-bgqm6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:19:36.779227 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:36.779191 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgqm6\" (UniqueName: \"kubernetes.io/projected/50d03554-8ebc-498c-848f-0f0dbc24dd41-kube-api-access-bgqm6\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:19:36.779227 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:36.779223 5437 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50d03554-8ebc-498c-848f-0f0dbc24dd41-plugins-config-volume\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:19:37.315754 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:37.315717 5437 generic.go:356] "Generic (PLEG): container finished" podID="50d03554-8ebc-498c-848f-0f0dbc24dd41" containerID="0183c1b9d6dbe402818660d32b905f2435403220dc6a2919210619070e3592d9" exitCode=0 Nov 03 02:19:37.315937 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:37.315767 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-50ca6d-774c6bf95-b26nc" Nov 03 02:19:37.315937 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:37.315803 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-50ca6d-774c6bf95-b26nc" event={"ID":"50d03554-8ebc-498c-848f-0f0dbc24dd41","Type":"ContainerDied","Data":"0183c1b9d6dbe402818660d32b905f2435403220dc6a2919210619070e3592d9"} Nov 03 02:19:37.315937 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:37.315864 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-50ca6d-774c6bf95-b26nc" event={"ID":"50d03554-8ebc-498c-848f-0f0dbc24dd41","Type":"ContainerDied","Data":"085f224b18575b19cc76da3186c67580a1f1416cae1db59db3c27677fff150c2"} Nov 03 02:19:37.315937 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:37.315902 5437 scope.go:122] "RemoveContainer" containerID="0183c1b9d6dbe402818660d32b905f2435403220dc6a2919210619070e3592d9" Nov 03 02:19:37.326533 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:37.326378 5437 scope.go:122] "RemoveContainer" containerID="0183c1b9d6dbe402818660d32b905f2435403220dc6a2919210619070e3592d9" Nov 03 02:19:37.326752 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:19:37.326668 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0183c1b9d6dbe402818660d32b905f2435403220dc6a2919210619070e3592d9\": container with ID starting with 0183c1b9d6dbe402818660d32b905f2435403220dc6a2919210619070e3592d9 not found: ID does not exist" containerID="0183c1b9d6dbe402818660d32b905f2435403220dc6a2919210619070e3592d9" Nov 03 02:19:37.326752 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:37.326697 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0183c1b9d6dbe402818660d32b905f2435403220dc6a2919210619070e3592d9"} err="failed to get container status \"0183c1b9d6dbe402818660d32b905f2435403220dc6a2919210619070e3592d9\": rpc error: code = NotFound desc = could not find container \"0183c1b9d6dbe402818660d32b905f2435403220dc6a2919210619070e3592d9\": container with ID starting with 0183c1b9d6dbe402818660d32b905f2435403220dc6a2919210619070e3592d9 not found: ID does not exist" Nov 03 02:19:37.486485 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:37.486453 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-50ca6d-774c6bf95-b26nc"] Nov 03 02:19:37.537811 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:37.537778 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-50ca6d-774c6bf95-b26nc"] Nov 03 02:19:38.091075 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:38.091043 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50d03554-8ebc-498c-848f-0f0dbc24dd41" path="/var/lib/kubelet/pods/50d03554-8ebc-498c-848f-0f0dbc24dd41/volumes" Nov 03 02:19:38.469399 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:38.469295 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:19:40.739042 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:40.739001 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:19:40.990452 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:40.990318 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:19:41.744602 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:41.744558 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:19:43.184929 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:43.184879 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:19:44.035821 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:19:44.035786 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 02:19:47.524778 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:47.524739 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:19:47.900199 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:47.900162 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:19:52.622897 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:52.622855 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:19:56.687248 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:56.687214 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:19:57.342662 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:19:57.342617 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:20:01.532505 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:20:01.532455 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:20:02.173208 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:20:02.173171 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:20:06.987827 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:20:06.987789 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:20:11.708281 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:20:11.708239 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:20:14.035477 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:20:14.035439 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 02:20:16.435214 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:20:16.435164 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:20:17.899859 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:20:17.899770 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:20:18.159137 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:20:18.159042 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:20:21.146266 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:20:21.146223 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:20:25.116354 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:20:25.116312 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:20:25.317047 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:20:25.317009 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:20:25.901341 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:20:25.901299 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:20:27.425560 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:20:27.425517 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:20:30.682334 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:20:30.682233 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:20:33.245997 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:20:33.245943 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:20:33.969917 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:20:33.969876 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-c70365-58bfdd8b7f-qw6f2"] Nov 03 02:20:34.280464 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:20:34.280349 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:20:34.481195 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:20:34.481156 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:20:35.514460 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:20:35.514416 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:20:35.775055 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:20:35.774963 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:20:36.051660 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:20:36.051630 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:20:38.970936 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:20:38.970872 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c70365-58bfdd8b7f-qw6f2" podUID="a5548a3f-3687-4453-ab8e-34f949a63e5a" containerName="payload-pre-processing" containerID="cri-o://807bad0567ade55ce480f0bfd61aed104c6679f8d46c55f85fcfa3faa380b073" gracePeriod=25 Nov 03 02:20:39.206720 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:20:39.206677 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:20:39.275207 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:20:39.275178 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c70365-58bfdd8b7f-qw6f2" Nov 03 02:20:39.465173 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:20:39.465139 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5548a3f-3687-4453-ab8e-34f949a63e5a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5548a3f-3687-4453-ab8e-34f949a63e5a-plugins-config-volume\") pod \"a5548a3f-3687-4453-ab8e-34f949a63e5a\" (UID: \"a5548a3f-3687-4453-ab8e-34f949a63e5a\") " Nov 03 02:20:39.465173 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:20:39.465180 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5548a3f-3687-4453-ab8e-34f949a63e5a-kube-api-access-s5w52\" (UniqueName: \"kubernetes.io/projected/a5548a3f-3687-4453-ab8e-34f949a63e5a-kube-api-access-s5w52\") pod \"a5548a3f-3687-4453-ab8e-34f949a63e5a\" (UID: \"a5548a3f-3687-4453-ab8e-34f949a63e5a\") " Nov 03 02:20:39.465588 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:20:39.465563 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5548a3f-3687-4453-ab8e-34f949a63e5a-plugins-config-volume" pod "a5548a3f-3687-4453-ab8e-34f949a63e5a" (UID: "a5548a3f-3687-4453-ab8e-34f949a63e5a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:20:39.467472 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:20:39.467445 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5548a3f-3687-4453-ab8e-34f949a63e5a-kube-api-access-s5w52" pod "a5548a3f-3687-4453-ab8e-34f949a63e5a" (UID: "a5548a3f-3687-4453-ab8e-34f949a63e5a"). InnerVolumeSpecName "kube-api-access-s5w52". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:20:39.566927 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:20:39.566896 5437 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5548a3f-3687-4453-ab8e-34f949a63e5a-plugins-config-volume\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:20:39.566927 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:20:39.566922 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5w52\" (UniqueName: \"kubernetes.io/projected/a5548a3f-3687-4453-ab8e-34f949a63e5a-kube-api-access-s5w52\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:20:39.723030 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:20:39.722994 5437 generic.go:356] "Generic (PLEG): container finished" podID="a5548a3f-3687-4453-ab8e-34f949a63e5a" containerID="807bad0567ade55ce480f0bfd61aed104c6679f8d46c55f85fcfa3faa380b073" exitCode=0 Nov 03 02:20:39.723228 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:20:39.723059 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c70365-58bfdd8b7f-qw6f2" Nov 03 02:20:39.723228 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:20:39.723080 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c70365-58bfdd8b7f-qw6f2" event={"ID":"a5548a3f-3687-4453-ab8e-34f949a63e5a","Type":"ContainerDied","Data":"807bad0567ade55ce480f0bfd61aed104c6679f8d46c55f85fcfa3faa380b073"} Nov 03 02:20:39.723228 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:20:39.723177 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c70365-58bfdd8b7f-qw6f2" event={"ID":"a5548a3f-3687-4453-ab8e-34f949a63e5a","Type":"ContainerDied","Data":"0b4af3fda6f78bfeef056329f0fc681867c7068cc29ea54b5e70e36e89e81803"} Nov 03 02:20:39.723228 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:20:39.723208 5437 scope.go:122] "RemoveContainer" containerID="807bad0567ade55ce480f0bfd61aed104c6679f8d46c55f85fcfa3faa380b073" Nov 03 02:20:39.733731 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:20:39.733712 5437 scope.go:122] "RemoveContainer" containerID="807bad0567ade55ce480f0bfd61aed104c6679f8d46c55f85fcfa3faa380b073" Nov 03 02:20:39.734016 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:20:39.733993 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"807bad0567ade55ce480f0bfd61aed104c6679f8d46c55f85fcfa3faa380b073\": container with ID starting with 807bad0567ade55ce480f0bfd61aed104c6679f8d46c55f85fcfa3faa380b073 not found: ID does not exist" containerID="807bad0567ade55ce480f0bfd61aed104c6679f8d46c55f85fcfa3faa380b073" Nov 03 02:20:39.734075 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:20:39.734024 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"807bad0567ade55ce480f0bfd61aed104c6679f8d46c55f85fcfa3faa380b073"} err="failed to get container status \"807bad0567ade55ce480f0bfd61aed104c6679f8d46c55f85fcfa3faa380b073\": rpc error: code = NotFound desc = could not find container \"807bad0567ade55ce480f0bfd61aed104c6679f8d46c55f85fcfa3faa380b073\": container with ID starting with 807bad0567ade55ce480f0bfd61aed104c6679f8d46c55f85fcfa3faa380b073 not found: ID does not exist" Nov 03 02:20:39.894693 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:20:39.894609 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-c70365-58bfdd8b7f-qw6f2"] Nov 03 02:20:39.945823 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:20:39.945792 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-c70365-58bfdd8b7f-qw6f2"] Nov 03 02:20:40.090473 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:20:40.090437 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5548a3f-3687-4453-ab8e-34f949a63e5a" path="/var/lib/kubelet/pods/a5548a3f-3687-4453-ab8e-34f949a63e5a/volumes" Nov 03 02:20:40.804714 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:20:40.804671 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:20:44.035937 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:20:44.035900 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 02:20:45.885982 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:20:45.885925 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:20:46.087579 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:20:46.087529 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:20:50.927061 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:20:50.927023 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:20:55.709118 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:20:55.709076 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:20:57.040428 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:20:57.040391 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:21:00.439865 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:21:00.439774 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:21:01.888305 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:21:01.888263 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:21:05.197686 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:21:05.197645 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:21:10.186046 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:21:10.186002 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:21:14.034716 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:21:14.034675 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 02:21:14.960019 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:21:14.959976 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:21:19.706997 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:21:19.706943 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:21:24.420738 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:21:24.420703 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:21:29.185609 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:21:29.185557 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:21:34.076086 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:21:34.076044 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:21:38.812817 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:21:38.812760 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:21:43.565979 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:21:43.565936 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:21:44.034747 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:21:44.034667 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 02:21:47.727755 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:21:47.727663 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:21:49.393340 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:21:49.393301 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:21:54.163949 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:21:54.163910 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:21:56.164734 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:21:56.164688 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:21:56.371073 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:21:56.371033 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:21:57.382503 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:21:57.382466 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:21:57.696756 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:21:57.696659 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:21:58.360685 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:21:58.360648 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:21:59.111043 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:21:59.111002 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:21:59.359908 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:21:59.359871 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:22:00.490013 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:00.489970 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:22:00.524765 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:00.524734 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-c34879-openshift-default-ccddfffff-f9pm9"] Nov 03 02:22:00.524978 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:00.524946 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-c34879-openshift-default-ccddfffff-f9pm9" podUID="f6c263e1-c60a-437a-a13a-579a420a833a" containerName="istio-proxy" containerID="cri-o://c023476daba1e99bcd170afb332c31e619826fccb9e6869a06b27bd3583fbf88" gracePeriod=30 Nov 03 02:22:02.242869 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:02.242831 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:22:04.343943 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:04.343909 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:22:05.597576 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:05.597532 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:22:05.799122 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:05.799090 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:22:05.813376 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:05.813343 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-c34879-openshift-default-ccddfffff-f9pm9" Nov 03 02:22:05.918235 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:05.918188 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6c263e1-c60a-437a-a13a-579a420a833a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6c263e1-c60a-437a-a13a-579a420a833a-istio-data\") pod \"f6c263e1-c60a-437a-a13a-579a420a833a\" (UID: \"f6c263e1-c60a-437a-a13a-579a420a833a\") " Nov 03 02:22:05.918442 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:05.918355 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6c263e1-c60a-437a-a13a-579a420a833a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6c263e1-c60a-437a-a13a-579a420a833a-workload-socket\") pod \"f6c263e1-c60a-437a-a13a-579a420a833a\" (UID: \"f6c263e1-c60a-437a-a13a-579a420a833a\") " Nov 03 02:22:05.918501 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:05.918471 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6c263e1-c60a-437a-a13a-579a420a833a-kube-api-access-4rlgf\" (UniqueName: \"kubernetes.io/projected/f6c263e1-c60a-437a-a13a-579a420a833a-kube-api-access-4rlgf\") pod \"f6c263e1-c60a-437a-a13a-579a420a833a\" (UID: \"f6c263e1-c60a-437a-a13a-579a420a833a\") " Nov 03 02:22:05.918539 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:05.918514 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6c263e1-c60a-437a-a13a-579a420a833a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6c263e1-c60a-437a-a13a-579a420a833a-workload-certs\") pod \"f6c263e1-c60a-437a-a13a-579a420a833a\" (UID: \"f6c263e1-c60a-437a-a13a-579a420a833a\") " Nov 03 02:22:05.918604 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:05.918575 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6c263e1-c60a-437a-a13a-579a420a833a-istio-data" pod "f6c263e1-c60a-437a-a13a-579a420a833a" (UID: "f6c263e1-c60a-437a-a13a-579a420a833a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:22:05.918656 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:05.918600 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6c263e1-c60a-437a-a13a-579a420a833a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6c263e1-c60a-437a-a13a-579a420a833a-credential-socket\") pod \"f6c263e1-c60a-437a-a13a-579a420a833a\" (UID: \"f6c263e1-c60a-437a-a13a-579a420a833a\") " Nov 03 02:22:05.918656 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:05.918641 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f6c263e1-c60a-437a-a13a-579a420a833a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6c263e1-c60a-437a-a13a-579a420a833a-istio-podinfo\") pod \"f6c263e1-c60a-437a-a13a-579a420a833a\" (UID: \"f6c263e1-c60a-437a-a13a-579a420a833a\") " Nov 03 02:22:05.918656 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:05.918588 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6c263e1-c60a-437a-a13a-579a420a833a-workload-socket" pod "f6c263e1-c60a-437a-a13a-579a420a833a" (UID: "f6c263e1-c60a-437a-a13a-579a420a833a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:22:05.918803 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:05.918696 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6c263e1-c60a-437a-a13a-579a420a833a-istio-token\" (UniqueName: \"kubernetes.io/projected/f6c263e1-c60a-437a-a13a-579a420a833a-istio-token\") pod \"f6c263e1-c60a-437a-a13a-579a420a833a\" (UID: \"f6c263e1-c60a-437a-a13a-579a420a833a\") " Nov 03 02:22:05.918867 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:05.918814 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6c263e1-c60a-437a-a13a-579a420a833a-workload-certs" pod "f6c263e1-c60a-437a-a13a-579a420a833a" (UID: "f6c263e1-c60a-437a-a13a-579a420a833a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:22:05.918867 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:05.918850 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6c263e1-c60a-437a-a13a-579a420a833a-credential-socket" pod "f6c263e1-c60a-437a-a13a-579a420a833a" (UID: "f6c263e1-c60a-437a-a13a-579a420a833a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:22:05.918867 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:05.918851 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6c263e1-c60a-437a-a13a-579a420a833a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6c263e1-c60a-437a-a13a-579a420a833a-istio-envoy\") pod \"f6c263e1-c60a-437a-a13a-579a420a833a\" (UID: \"f6c263e1-c60a-437a-a13a-579a420a833a\") " Nov 03 02:22:05.919051 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:05.918903 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6c263e1-c60a-437a-a13a-579a420a833a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6c263e1-c60a-437a-a13a-579a420a833a-istiod-ca-cert\") pod \"f6c263e1-c60a-437a-a13a-579a420a833a\" (UID: \"f6c263e1-c60a-437a-a13a-579a420a833a\") " Nov 03 02:22:05.919280 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:05.919257 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6c263e1-c60a-437a-a13a-579a420a833a-istiod-ca-cert" pod "f6c263e1-c60a-437a-a13a-579a420a833a" (UID: "f6c263e1-c60a-437a-a13a-579a420a833a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:22:05.919599 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:05.919576 5437 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6c263e1-c60a-437a-a13a-579a420a833a-istio-data\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:22:05.919689 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:05.919604 5437 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6c263e1-c60a-437a-a13a-579a420a833a-workload-socket\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:22:05.919689 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:05.919621 5437 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6c263e1-c60a-437a-a13a-579a420a833a-workload-certs\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:22:05.919689 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:05.919636 5437 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6c263e1-c60a-437a-a13a-579a420a833a-credential-socket\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:22:05.919689 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:05.919651 5437 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6c263e1-c60a-437a-a13a-579a420a833a-istiod-ca-cert\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:22:05.921183 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:05.921152 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6c263e1-c60a-437a-a13a-579a420a833a-istio-token" pod "f6c263e1-c60a-437a-a13a-579a420a833a" (UID: "f6c263e1-c60a-437a-a13a-579a420a833a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:22:05.921285 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:05.921200 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f6c263e1-c60a-437a-a13a-579a420a833a-istio-podinfo" pod "f6c263e1-c60a-437a-a13a-579a420a833a" (UID: "f6c263e1-c60a-437a-a13a-579a420a833a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:22:05.921285 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:05.921230 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6c263e1-c60a-437a-a13a-579a420a833a-kube-api-access-4rlgf" pod "f6c263e1-c60a-437a-a13a-579a420a833a" (UID: "f6c263e1-c60a-437a-a13a-579a420a833a"). InnerVolumeSpecName "kube-api-access-4rlgf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:22:05.921439 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:05.921422 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6c263e1-c60a-437a-a13a-579a420a833a-istio-envoy" pod "f6c263e1-c60a-437a-a13a-579a420a833a" (UID: "f6c263e1-c60a-437a-a13a-579a420a833a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:22:06.006433 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:06.006392 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:22:06.021042 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:06.021007 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rlgf\" (UniqueName: \"kubernetes.io/projected/f6c263e1-c60a-437a-a13a-579a420a833a-kube-api-access-4rlgf\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:22:06.021042 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:06.021038 5437 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6c263e1-c60a-437a-a13a-579a420a833a-istio-podinfo\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:22:06.021042 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:06.021048 5437 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6c263e1-c60a-437a-a13a-579a420a833a-istio-token\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:22:06.021348 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:06.021059 5437 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6c263e1-c60a-437a-a13a-579a420a833a-istio-envoy\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:22:06.210396 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:06.210341 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:22:06.274297 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:06.274261 5437 generic.go:356] "Generic (PLEG): container finished" podID="f6c263e1-c60a-437a-a13a-579a420a833a" containerID="c023476daba1e99bcd170afb332c31e619826fccb9e6869a06b27bd3583fbf88" exitCode=0 Nov 03 02:22:06.274498 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:06.274339 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-c34879-openshift-default-ccddfffff-f9pm9" Nov 03 02:22:06.274498 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:06.274339 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-c34879-openshift-default-ccddfffff-f9pm9" event={"ID":"f6c263e1-c60a-437a-a13a-579a420a833a","Type":"ContainerDied","Data":"c023476daba1e99bcd170afb332c31e619826fccb9e6869a06b27bd3583fbf88"} Nov 03 02:22:06.274498 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:06.274432 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-c34879-openshift-default-ccddfffff-f9pm9" event={"ID":"f6c263e1-c60a-437a-a13a-579a420a833a","Type":"ContainerDied","Data":"ec7f323aff081adee6a12169746f35040d72c9bfff2b2bd81df835d73883776d"} Nov 03 02:22:06.274498 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:06.274466 5437 scope.go:122] "RemoveContainer" containerID="c023476daba1e99bcd170afb332c31e619826fccb9e6869a06b27bd3583fbf88" Nov 03 02:22:06.285911 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:06.285888 5437 scope.go:122] "RemoveContainer" containerID="c023476daba1e99bcd170afb332c31e619826fccb9e6869a06b27bd3583fbf88" Nov 03 02:22:06.286196 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:22:06.286169 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c023476daba1e99bcd170afb332c31e619826fccb9e6869a06b27bd3583fbf88\": container with ID starting with c023476daba1e99bcd170afb332c31e619826fccb9e6869a06b27bd3583fbf88 not found: ID does not exist" containerID="c023476daba1e99bcd170afb332c31e619826fccb9e6869a06b27bd3583fbf88" Nov 03 02:22:06.286279 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:06.286210 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c023476daba1e99bcd170afb332c31e619826fccb9e6869a06b27bd3583fbf88"} err="failed to get container status \"c023476daba1e99bcd170afb332c31e619826fccb9e6869a06b27bd3583fbf88\": rpc error: code = NotFound desc = could not find container \"c023476daba1e99bcd170afb332c31e619826fccb9e6869a06b27bd3583fbf88\": container with ID starting with c023476daba1e99bcd170afb332c31e619826fccb9e6869a06b27bd3583fbf88 not found: ID does not exist" Nov 03 02:22:06.446160 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:06.446075 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-c34879-openshift-default-ccddfffff-f9pm9"] Nov 03 02:22:06.497076 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:06.497044 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-c34879-openshift-default-ccddfffff-f9pm9"] Nov 03 02:22:06.935989 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:06.935957 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:22:07.567919 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:07.567886 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-bd872a-7b5f4b9979-g4cqr"] Nov 03 02:22:07.877952 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:07.877845 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:22:08.075408 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:08.075376 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:22:08.089270 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:08.089242 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6c263e1-c60a-437a-a13a-579a420a833a" path="/var/lib/kubelet/pods/f6c263e1-c60a-437a-a13a-579a420a833a/volumes" Nov 03 02:22:09.313229 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:09.313189 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:22:12.433795 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:12.433739 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:22:12.569034 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:12.568993 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-bd872a-7b5f4b9979-g4cqr" podUID="32db9a05-1703-4fbc-b999-fdba1981f184" containerName="payload-pre-processing" containerID="cri-o://1f8ff2b0014afd456823ac4d9b24d45cecdca1b3c11b2ff2fcf612af6ecf5f6b" gracePeriod=25 Nov 03 02:22:12.649292 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:12.649255 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-aab75e-94fdc78bf-x7ztb"] Nov 03 02:22:12.680597 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:12.680555 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:22:12.757870 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:12.757836 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-aab75e-d78bd8b87-5wws6"] Nov 03 02:22:12.832716 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:12.832687 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-bd872a-7b5f4b9979-g4cqr" Nov 03 02:22:12.908917 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:12.908880 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32db9a05-1703-4fbc-b999-fdba1981f184-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32db9a05-1703-4fbc-b999-fdba1981f184-plugins-config-volume\") pod \"32db9a05-1703-4fbc-b999-fdba1981f184\" (UID: \"32db9a05-1703-4fbc-b999-fdba1981f184\") " Nov 03 02:22:12.908917 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:12.908923 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32db9a05-1703-4fbc-b999-fdba1981f184-kube-api-access-4hhgp\" (UniqueName: \"kubernetes.io/projected/32db9a05-1703-4fbc-b999-fdba1981f184-kube-api-access-4hhgp\") pod \"32db9a05-1703-4fbc-b999-fdba1981f184\" (UID: \"32db9a05-1703-4fbc-b999-fdba1981f184\") " Nov 03 02:22:12.909331 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:12.909308 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32db9a05-1703-4fbc-b999-fdba1981f184-plugins-config-volume" pod "32db9a05-1703-4fbc-b999-fdba1981f184" (UID: "32db9a05-1703-4fbc-b999-fdba1981f184"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:22:12.909541 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:12.909524 5437 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32db9a05-1703-4fbc-b999-fdba1981f184-plugins-config-volume\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:22:12.911108 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:12.911081 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32db9a05-1703-4fbc-b999-fdba1981f184-kube-api-access-4hhgp" pod "32db9a05-1703-4fbc-b999-fdba1981f184" (UID: "32db9a05-1703-4fbc-b999-fdba1981f184"). InnerVolumeSpecName "kube-api-access-4hhgp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:22:12.958040 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:12.957951 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:22:13.010456 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:13.010424 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hhgp\" (UniqueName: \"kubernetes.io/projected/32db9a05-1703-4fbc-b999-fdba1981f184-kube-api-access-4hhgp\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:22:13.159908 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:13.159862 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:22:13.324307 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:13.324278 5437 generic.go:356] "Generic (PLEG): container finished" podID="32db9a05-1703-4fbc-b999-fdba1981f184" containerID="1f8ff2b0014afd456823ac4d9b24d45cecdca1b3c11b2ff2fcf612af6ecf5f6b" exitCode=0 Nov 03 02:22:13.324539 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:13.324330 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-bd872a-7b5f4b9979-g4cqr" event={"ID":"32db9a05-1703-4fbc-b999-fdba1981f184","Type":"ContainerDied","Data":"1f8ff2b0014afd456823ac4d9b24d45cecdca1b3c11b2ff2fcf612af6ecf5f6b"} Nov 03 02:22:13.324539 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:13.324346 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-bd872a-7b5f4b9979-g4cqr" Nov 03 02:22:13.324539 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:13.324384 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-bd872a-7b5f4b9979-g4cqr" event={"ID":"32db9a05-1703-4fbc-b999-fdba1981f184","Type":"ContainerDied","Data":"5a7bdc287915f3ab024cb6ce81c6cfd0c51b1f7ec22d796cf22b9744e7469f54"} Nov 03 02:22:13.324539 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:13.324408 5437 scope.go:122] "RemoveContainer" containerID="1f8ff2b0014afd456823ac4d9b24d45cecdca1b3c11b2ff2fcf612af6ecf5f6b" Nov 03 02:22:13.336946 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:13.336928 5437 scope.go:122] "RemoveContainer" containerID="1f8ff2b0014afd456823ac4d9b24d45cecdca1b3c11b2ff2fcf612af6ecf5f6b" Nov 03 02:22:13.337241 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:22:13.337217 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f8ff2b0014afd456823ac4d9b24d45cecdca1b3c11b2ff2fcf612af6ecf5f6b\": container with ID starting with 1f8ff2b0014afd456823ac4d9b24d45cecdca1b3c11b2ff2fcf612af6ecf5f6b not found: ID does not exist" containerID="1f8ff2b0014afd456823ac4d9b24d45cecdca1b3c11b2ff2fcf612af6ecf5f6b" Nov 03 02:22:13.337280 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:13.337250 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f8ff2b0014afd456823ac4d9b24d45cecdca1b3c11b2ff2fcf612af6ecf5f6b"} err="failed to get container status \"1f8ff2b0014afd456823ac4d9b24d45cecdca1b3c11b2ff2fcf612af6ecf5f6b\": rpc error: code = NotFound desc = could not find container \"1f8ff2b0014afd456823ac4d9b24d45cecdca1b3c11b2ff2fcf612af6ecf5f6b\": container with ID starting with 1f8ff2b0014afd456823ac4d9b24d45cecdca1b3c11b2ff2fcf612af6ecf5f6b not found: ID does not exist" Nov 03 02:22:13.496671 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:13.496636 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-bd872a-7b5f4b9979-g4cqr"] Nov 03 02:22:13.501550 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:13.501521 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-bd872a-7b5f4b9979-g4cqr"] Nov 03 02:22:14.038092 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:22:14.038052 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 02:22:14.089232 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:14.089198 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32db9a05-1703-4fbc-b999-fdba1981f184" path="/var/lib/kubelet/pods/32db9a05-1703-4fbc-b999-fdba1981f184/volumes" Nov 03 02:22:14.397330 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:14.397285 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:22:17.650440 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:17.650391 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-aab75e-94fdc78bf-x7ztb" podUID="97251b61-207f-4a1a-ad2c-89e111f4c38d" containerName="payload-processing" containerID="cri-o://8ea3a0aa8cf8dbbe71336fe4eff556db264abcdd34c488e8abf064b00159a691" gracePeriod=25 Nov 03 02:22:17.758830 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:17.758773 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-aab75e-d78bd8b87-5wws6" podUID="b6687078-b2c2-4a89-9bc9-d0fa069ce917" containerName="payload-pre-processing" containerID="cri-o://594efcb70140b4fa8c9afd8d4ae97e8226dca6038103ab7a0b059053ed536524" gracePeriod=25 Nov 03 02:22:17.934946 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:17.934919 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-aab75e-94fdc78bf-x7ztb" Nov 03 02:22:18.029981 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:18.029956 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-aab75e-d78bd8b87-5wws6" Nov 03 02:22:18.073811 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:18.073781 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97251b61-207f-4a1a-ad2c-89e111f4c38d-kube-api-access-mcrfx\" (UniqueName: \"kubernetes.io/projected/97251b61-207f-4a1a-ad2c-89e111f4c38d-kube-api-access-mcrfx\") pod \"97251b61-207f-4a1a-ad2c-89e111f4c38d\" (UID: \"97251b61-207f-4a1a-ad2c-89e111f4c38d\") " Nov 03 02:22:18.073977 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:18.073836 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97251b61-207f-4a1a-ad2c-89e111f4c38d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97251b61-207f-4a1a-ad2c-89e111f4c38d-plugins-config-volume\") pod \"97251b61-207f-4a1a-ad2c-89e111f4c38d\" (UID: \"97251b61-207f-4a1a-ad2c-89e111f4c38d\") " Nov 03 02:22:18.074284 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:18.074256 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97251b61-207f-4a1a-ad2c-89e111f4c38d-plugins-config-volume" pod "97251b61-207f-4a1a-ad2c-89e111f4c38d" (UID: "97251b61-207f-4a1a-ad2c-89e111f4c38d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:22:18.074429 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:18.074378 5437 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97251b61-207f-4a1a-ad2c-89e111f4c38d-plugins-config-volume\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:22:18.075878 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:18.075861 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97251b61-207f-4a1a-ad2c-89e111f4c38d-kube-api-access-mcrfx" pod "97251b61-207f-4a1a-ad2c-89e111f4c38d" (UID: "97251b61-207f-4a1a-ad2c-89e111f4c38d"). InnerVolumeSpecName "kube-api-access-mcrfx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:22:18.175223 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:18.175132 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6687078-b2c2-4a89-9bc9-d0fa069ce917-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6687078-b2c2-4a89-9bc9-d0fa069ce917-plugins-config-volume\") pod \"b6687078-b2c2-4a89-9bc9-d0fa069ce917\" (UID: \"b6687078-b2c2-4a89-9bc9-d0fa069ce917\") " Nov 03 02:22:18.175442 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:18.175341 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6687078-b2c2-4a89-9bc9-d0fa069ce917-kube-api-access-r8vmr\" (UniqueName: \"kubernetes.io/projected/b6687078-b2c2-4a89-9bc9-d0fa069ce917-kube-api-access-r8vmr\") pod \"b6687078-b2c2-4a89-9bc9-d0fa069ce917\" (UID: \"b6687078-b2c2-4a89-9bc9-d0fa069ce917\") " Nov 03 02:22:18.175587 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:18.175558 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6687078-b2c2-4a89-9bc9-d0fa069ce917-plugins-config-volume" pod "b6687078-b2c2-4a89-9bc9-d0fa069ce917" (UID: "b6687078-b2c2-4a89-9bc9-d0fa069ce917"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:22:18.175859 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:18.175839 5437 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6687078-b2c2-4a89-9bc9-d0fa069ce917-plugins-config-volume\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:22:18.175923 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:18.175865 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mcrfx\" (UniqueName: \"kubernetes.io/projected/97251b61-207f-4a1a-ad2c-89e111f4c38d-kube-api-access-mcrfx\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:22:18.177568 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:18.177537 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6687078-b2c2-4a89-9bc9-d0fa069ce917-kube-api-access-r8vmr" pod "b6687078-b2c2-4a89-9bc9-d0fa069ce917" (UID: "b6687078-b2c2-4a89-9bc9-d0fa069ce917"). InnerVolumeSpecName "kube-api-access-r8vmr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:22:18.277317 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:18.277278 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8vmr\" (UniqueName: \"kubernetes.io/projected/b6687078-b2c2-4a89-9bc9-d0fa069ce917-kube-api-access-r8vmr\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:22:18.340619 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:18.340581 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:22:18.362228 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:18.362191 5437 generic.go:356] "Generic (PLEG): container finished" podID="97251b61-207f-4a1a-ad2c-89e111f4c38d" containerID="8ea3a0aa8cf8dbbe71336fe4eff556db264abcdd34c488e8abf064b00159a691" exitCode=0 Nov 03 02:22:18.362424 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:18.362243 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-aab75e-94fdc78bf-x7ztb" Nov 03 02:22:18.362424 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:18.362265 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-aab75e-94fdc78bf-x7ztb" event={"ID":"97251b61-207f-4a1a-ad2c-89e111f4c38d","Type":"ContainerDied","Data":"8ea3a0aa8cf8dbbe71336fe4eff556db264abcdd34c488e8abf064b00159a691"} Nov 03 02:22:18.362424 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:18.362323 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-aab75e-94fdc78bf-x7ztb" event={"ID":"97251b61-207f-4a1a-ad2c-89e111f4c38d","Type":"ContainerDied","Data":"3ccde67baf79e943b783f39817a7d152b0ab4770a3b50e71ddb9f496717be097"} Nov 03 02:22:18.362424 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:18.362343 5437 scope.go:122] "RemoveContainer" containerID="8ea3a0aa8cf8dbbe71336fe4eff556db264abcdd34c488e8abf064b00159a691" Nov 03 02:22:18.363838 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:18.363816 5437 generic.go:356] "Generic (PLEG): container finished" podID="b6687078-b2c2-4a89-9bc9-d0fa069ce917" containerID="594efcb70140b4fa8c9afd8d4ae97e8226dca6038103ab7a0b059053ed536524" exitCode=0 Nov 03 02:22:18.363942 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:18.363860 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-aab75e-d78bd8b87-5wws6" Nov 03 02:22:18.363942 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:18.363897 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-aab75e-d78bd8b87-5wws6" event={"ID":"b6687078-b2c2-4a89-9bc9-d0fa069ce917","Type":"ContainerDied","Data":"594efcb70140b4fa8c9afd8d4ae97e8226dca6038103ab7a0b059053ed536524"} Nov 03 02:22:18.363942 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:18.363934 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-aab75e-d78bd8b87-5wws6" event={"ID":"b6687078-b2c2-4a89-9bc9-d0fa069ce917","Type":"ContainerDied","Data":"a96195fa3e586a06d3f5637007d48d4ae8549e8dfd1b18fde94cb641b711a7ce"} Nov 03 02:22:18.373301 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:18.373283 5437 scope.go:122] "RemoveContainer" containerID="8ea3a0aa8cf8dbbe71336fe4eff556db264abcdd34c488e8abf064b00159a691" Nov 03 02:22:18.373618 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:22:18.373600 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ea3a0aa8cf8dbbe71336fe4eff556db264abcdd34c488e8abf064b00159a691\": container with ID starting with 8ea3a0aa8cf8dbbe71336fe4eff556db264abcdd34c488e8abf064b00159a691 not found: ID does not exist" containerID="8ea3a0aa8cf8dbbe71336fe4eff556db264abcdd34c488e8abf064b00159a691" Nov 03 02:22:18.373677 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:18.373628 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ea3a0aa8cf8dbbe71336fe4eff556db264abcdd34c488e8abf064b00159a691"} err="failed to get container status \"8ea3a0aa8cf8dbbe71336fe4eff556db264abcdd34c488e8abf064b00159a691\": rpc error: code = NotFound desc = could not find container \"8ea3a0aa8cf8dbbe71336fe4eff556db264abcdd34c488e8abf064b00159a691\": container with ID starting with 8ea3a0aa8cf8dbbe71336fe4eff556db264abcdd34c488e8abf064b00159a691 not found: ID does not exist" Nov 03 02:22:18.373677 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:18.373648 5437 scope.go:122] "RemoveContainer" containerID="594efcb70140b4fa8c9afd8d4ae97e8226dca6038103ab7a0b059053ed536524" Nov 03 02:22:18.382358 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:18.382339 5437 scope.go:122] "RemoveContainer" containerID="594efcb70140b4fa8c9afd8d4ae97e8226dca6038103ab7a0b059053ed536524" Nov 03 02:22:18.382669 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:22:18.382647 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"594efcb70140b4fa8c9afd8d4ae97e8226dca6038103ab7a0b059053ed536524\": container with ID starting with 594efcb70140b4fa8c9afd8d4ae97e8226dca6038103ab7a0b059053ed536524 not found: ID does not exist" containerID="594efcb70140b4fa8c9afd8d4ae97e8226dca6038103ab7a0b059053ed536524" Nov 03 02:22:18.382721 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:18.382677 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"594efcb70140b4fa8c9afd8d4ae97e8226dca6038103ab7a0b059053ed536524"} err="failed to get container status \"594efcb70140b4fa8c9afd8d4ae97e8226dca6038103ab7a0b059053ed536524\": rpc error: code = NotFound desc = could not find container \"594efcb70140b4fa8c9afd8d4ae97e8226dca6038103ab7a0b059053ed536524\": container with ID starting with 594efcb70140b4fa8c9afd8d4ae97e8226dca6038103ab7a0b059053ed536524 not found: ID does not exist" Nov 03 02:22:18.531896 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:18.531817 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-aab75e-94fdc78bf-x7ztb"] Nov 03 02:22:18.536382 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:18.536340 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-aab75e-94fdc78bf-x7ztb"] Nov 03 02:22:18.655456 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:18.655424 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-aab75e-d78bd8b87-5wws6"] Nov 03 02:22:18.660771 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:18.660743 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-aab75e-d78bd8b87-5wws6"] Nov 03 02:22:19.483695 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:19.483642 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:22:20.088985 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:20.088948 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97251b61-207f-4a1a-ad2c-89e111f4c38d" path="/var/lib/kubelet/pods/97251b61-207f-4a1a-ad2c-89e111f4c38d/volumes" Nov 03 02:22:20.089443 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:20.089427 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6687078-b2c2-4a89-9bc9-d0fa069ce917" path="/var/lib/kubelet/pods/b6687078-b2c2-4a89-9bc9-d0fa069ce917/volumes" Nov 03 02:22:24.650197 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:24.650158 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:22:29.084178 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:29.084127 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:22:30.786106 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:30.786013 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:22:31.041835 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:31.041741 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:22:31.235538 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:31.235496 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-50ca6d-openshift-default-6b7b98d8c7-499js"] Nov 03 02:22:31.235758 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:31.235735 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-50ca6d-openshift-default-6b7b98d8c7-499js" podUID="6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea" containerName="istio-proxy" containerID="cri-o://69f899906e8c170453bf9a7a42d26165c8df0a3ac53a5a98b28acb4d21b7cd47" gracePeriod=30 Nov 03 02:22:31.296540 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:31.296446 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:22:33.036505 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:33.036464 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:22:33.309997 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:33.309956 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:22:34.038214 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:34.038179 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-ea5c75-openshift-default-75dc4675bf-qbhjq"] Nov 03 02:22:34.038800 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:34.038786 5437 cpu_manager.go:414] "containerMap: removing container" podUID="50d03554-8ebc-498c-848f-0f0dbc24dd41" containerName="payload-processing" Nov 03 02:22:34.038847 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:34.038801 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50d03554-8ebc-498c-848f-0f0dbc24dd41" containerName="payload-processing" Nov 03 02:22:34.038847 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:34.038814 5437 cpu_manager.go:414] "containerMap: removing container" podUID="f6c263e1-c60a-437a-a13a-579a420a833a" containerName="istio-proxy" Nov 03 02:22:34.038847 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:34.038819 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6c263e1-c60a-437a-a13a-579a420a833a" containerName="istio-proxy" Nov 03 02:22:34.038847 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:34.038834 5437 cpu_manager.go:414] "containerMap: removing container" podUID="b6687078-b2c2-4a89-9bc9-d0fa069ce917" containerName="payload-pre-processing" Nov 03 02:22:34.038847 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:34.038840 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6687078-b2c2-4a89-9bc9-d0fa069ce917" containerName="payload-pre-processing" Nov 03 02:22:34.039033 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:34.038853 5437 cpu_manager.go:414] "containerMap: removing container" podUID="a5548a3f-3687-4453-ab8e-34f949a63e5a" containerName="payload-pre-processing" Nov 03 02:22:34.039033 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:34.038858 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5548a3f-3687-4453-ab8e-34f949a63e5a" containerName="payload-pre-processing" Nov 03 02:22:34.039033 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:34.038864 5437 cpu_manager.go:414] "containerMap: removing container" podUID="97251b61-207f-4a1a-ad2c-89e111f4c38d" containerName="payload-processing" Nov 03 02:22:34.039033 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:34.038868 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97251b61-207f-4a1a-ad2c-89e111f4c38d" containerName="payload-processing" Nov 03 02:22:34.039033 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:34.038887 5437 cpu_manager.go:414] "containerMap: removing container" podUID="32db9a05-1703-4fbc-b999-fdba1981f184" containerName="payload-pre-processing" Nov 03 02:22:34.039033 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:34.038892 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32db9a05-1703-4fbc-b999-fdba1981f184" containerName="payload-pre-processing" Nov 03 02:22:34.039033 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:34.038981 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="50d03554-8ebc-498c-848f-0f0dbc24dd41" containerName="payload-processing" Nov 03 02:22:34.039033 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:34.038991 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6c263e1-c60a-437a-a13a-579a420a833a" containerName="istio-proxy" Nov 03 02:22:34.039033 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:34.038998 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6687078-b2c2-4a89-9bc9-d0fa069ce917" containerName="payload-pre-processing" Nov 03 02:22:34.039033 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:34.039005 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5548a3f-3687-4453-ab8e-34f949a63e5a" containerName="payload-pre-processing" Nov 03 02:22:34.039033 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:34.039011 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="97251b61-207f-4a1a-ad2c-89e111f4c38d" containerName="payload-processing" Nov 03 02:22:34.039033 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:34.039019 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="32db9a05-1703-4fbc-b999-fdba1981f184" containerName="payload-pre-processing" Nov 03 02:22:34.044253 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:34.044233 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-ea5c75-openshift-default-75dc4675bf-qbhjq" Nov 03 02:22:34.103812 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:34.103782 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-ea5c75-openshift-default-dockercfg-zpvmg\"" Nov 03 02:22:34.153486 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:34.153447 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-ea5c75-openshift-default-75dc4675bf-qbhjq"] Nov 03 02:22:34.170052 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:34.170015 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jb294\" (UniqueName: \"kubernetes.io/projected/38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c-kube-api-access-jb294\") pod \"e2e-extmod-a-ea5c75-openshift-default-75dc4675bf-qbhjq\" (UID: \"38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c\") " pod="openshift-ingress/e2e-extmod-a-ea5c75-openshift-default-75dc4675bf-qbhjq" Nov 03 02:22:34.170235 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:34.170113 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c-istio-token\") pod \"e2e-extmod-a-ea5c75-openshift-default-75dc4675bf-qbhjq\" (UID: \"38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c\") " pod="openshift-ingress/e2e-extmod-a-ea5c75-openshift-default-75dc4675bf-qbhjq" Nov 03 02:22:34.170327 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:34.170301 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c-workload-socket\") pod \"e2e-extmod-a-ea5c75-openshift-default-75dc4675bf-qbhjq\" (UID: \"38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c\") " pod="openshift-ingress/e2e-extmod-a-ea5c75-openshift-default-75dc4675bf-qbhjq" Nov 03 02:22:34.170427 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:34.170345 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c-istio-data\") pod \"e2e-extmod-a-ea5c75-openshift-default-75dc4675bf-qbhjq\" (UID: \"38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c\") " pod="openshift-ingress/e2e-extmod-a-ea5c75-openshift-default-75dc4675bf-qbhjq" Nov 03 02:22:34.171040 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:34.171009 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c-istiod-ca-cert\") pod \"e2e-extmod-a-ea5c75-openshift-default-75dc4675bf-qbhjq\" (UID: \"38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c\") " pod="openshift-ingress/e2e-extmod-a-ea5c75-openshift-default-75dc4675bf-qbhjq" Nov 03 02:22:34.171391 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:34.171186 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c-istio-envoy\") pod \"e2e-extmod-a-ea5c75-openshift-default-75dc4675bf-qbhjq\" (UID: \"38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c\") " pod="openshift-ingress/e2e-extmod-a-ea5c75-openshift-default-75dc4675bf-qbhjq" Nov 03 02:22:34.171391 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:34.171236 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c-istio-podinfo\") pod \"e2e-extmod-a-ea5c75-openshift-default-75dc4675bf-qbhjq\" (UID: \"38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c\") " pod="openshift-ingress/e2e-extmod-a-ea5c75-openshift-default-75dc4675bf-qbhjq" Nov 03 02:22:34.171391 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:34.171275 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c-workload-certs\") pod \"e2e-extmod-a-ea5c75-openshift-default-75dc4675bf-qbhjq\" (UID: \"38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c\") " pod="openshift-ingress/e2e-extmod-a-ea5c75-openshift-default-75dc4675bf-qbhjq" Nov 03 02:22:34.171391 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:34.171332 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c-credential-socket\") pod \"e2e-extmod-a-ea5c75-openshift-default-75dc4675bf-qbhjq\" (UID: \"38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c\") " pod="openshift-ingress/e2e-extmod-a-ea5c75-openshift-default-75dc4675bf-qbhjq" Nov 03 02:22:34.182414 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:34.182380 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:22:34.273164 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:34.273128 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jb294\" (UniqueName: \"kubernetes.io/projected/38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c-kube-api-access-jb294\") pod \"e2e-extmod-a-ea5c75-openshift-default-75dc4675bf-qbhjq\" (UID: \"38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c\") " pod="openshift-ingress/e2e-extmod-a-ea5c75-openshift-default-75dc4675bf-qbhjq" Nov 03 02:22:34.273402 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:34.273182 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c-istio-token\") pod \"e2e-extmod-a-ea5c75-openshift-default-75dc4675bf-qbhjq\" (UID: \"38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c\") " pod="openshift-ingress/e2e-extmod-a-ea5c75-openshift-default-75dc4675bf-qbhjq" Nov 03 02:22:34.273402 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:34.273218 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c-workload-socket\") pod \"e2e-extmod-a-ea5c75-openshift-default-75dc4675bf-qbhjq\" (UID: \"38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c\") " pod="openshift-ingress/e2e-extmod-a-ea5c75-openshift-default-75dc4675bf-qbhjq" Nov 03 02:22:34.273402 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:34.273235 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c-istio-data\") pod \"e2e-extmod-a-ea5c75-openshift-default-75dc4675bf-qbhjq\" (UID: \"38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c\") " pod="openshift-ingress/e2e-extmod-a-ea5c75-openshift-default-75dc4675bf-qbhjq" Nov 03 02:22:34.273402 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:34.273263 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c-istiod-ca-cert\") pod \"e2e-extmod-a-ea5c75-openshift-default-75dc4675bf-qbhjq\" (UID: \"38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c\") " pod="openshift-ingress/e2e-extmod-a-ea5c75-openshift-default-75dc4675bf-qbhjq" Nov 03 02:22:34.273402 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:34.273336 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c-istio-envoy\") pod \"e2e-extmod-a-ea5c75-openshift-default-75dc4675bf-qbhjq\" (UID: \"38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c\") " pod="openshift-ingress/e2e-extmod-a-ea5c75-openshift-default-75dc4675bf-qbhjq" Nov 03 02:22:34.273402 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:34.273403 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c-istio-podinfo\") pod \"e2e-extmod-a-ea5c75-openshift-default-75dc4675bf-qbhjq\" (UID: \"38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c\") " pod="openshift-ingress/e2e-extmod-a-ea5c75-openshift-default-75dc4675bf-qbhjq" Nov 03 02:22:34.273766 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:34.273436 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c-workload-certs\") pod \"e2e-extmod-a-ea5c75-openshift-default-75dc4675bf-qbhjq\" (UID: \"38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c\") " pod="openshift-ingress/e2e-extmod-a-ea5c75-openshift-default-75dc4675bf-qbhjq" Nov 03 02:22:34.273766 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:34.273472 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c-credential-socket\") pod \"e2e-extmod-a-ea5c75-openshift-default-75dc4675bf-qbhjq\" (UID: \"38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c\") " pod="openshift-ingress/e2e-extmod-a-ea5c75-openshift-default-75dc4675bf-qbhjq" Nov 03 02:22:34.273766 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:34.273722 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c-workload-socket\") pod \"e2e-extmod-a-ea5c75-openshift-default-75dc4675bf-qbhjq\" (UID: \"38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c\") " pod="openshift-ingress/e2e-extmod-a-ea5c75-openshift-default-75dc4675bf-qbhjq" Nov 03 02:22:34.273766 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:34.273755 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c-istio-data\") pod \"e2e-extmod-a-ea5c75-openshift-default-75dc4675bf-qbhjq\" (UID: \"38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c\") " pod="openshift-ingress/e2e-extmod-a-ea5c75-openshift-default-75dc4675bf-qbhjq" Nov 03 02:22:34.274053 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:34.273875 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c-workload-certs\") pod \"e2e-extmod-a-ea5c75-openshift-default-75dc4675bf-qbhjq\" (UID: \"38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c\") " pod="openshift-ingress/e2e-extmod-a-ea5c75-openshift-default-75dc4675bf-qbhjq" Nov 03 02:22:34.274053 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:34.273889 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c-credential-socket\") pod \"e2e-extmod-a-ea5c75-openshift-default-75dc4675bf-qbhjq\" (UID: \"38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c\") " pod="openshift-ingress/e2e-extmod-a-ea5c75-openshift-default-75dc4675bf-qbhjq" Nov 03 02:22:34.274276 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:34.274253 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c-istiod-ca-cert\") pod \"e2e-extmod-a-ea5c75-openshift-default-75dc4675bf-qbhjq\" (UID: \"38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c\") " pod="openshift-ingress/e2e-extmod-a-ea5c75-openshift-default-75dc4675bf-qbhjq" Nov 03 02:22:34.275771 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:34.275749 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c-istio-envoy\") pod \"e2e-extmod-a-ea5c75-openshift-default-75dc4675bf-qbhjq\" (UID: \"38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c\") " pod="openshift-ingress/e2e-extmod-a-ea5c75-openshift-default-75dc4675bf-qbhjq" Nov 03 02:22:34.275987 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:34.275968 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c-istio-podinfo\") pod \"e2e-extmod-a-ea5c75-openshift-default-75dc4675bf-qbhjq\" (UID: \"38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c\") " pod="openshift-ingress/e2e-extmod-a-ea5c75-openshift-default-75dc4675bf-qbhjq" Nov 03 02:22:34.336234 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:34.336200 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c-istio-token\") pod \"e2e-extmod-a-ea5c75-openshift-default-75dc4675bf-qbhjq\" (UID: \"38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c\") " pod="openshift-ingress/e2e-extmod-a-ea5c75-openshift-default-75dc4675bf-qbhjq" Nov 03 02:22:34.336440 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:34.336410 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jb294\" (UniqueName: \"kubernetes.io/projected/38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c-kube-api-access-jb294\") pod \"e2e-extmod-a-ea5c75-openshift-default-75dc4675bf-qbhjq\" (UID: \"38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c\") " pod="openshift-ingress/e2e-extmod-a-ea5c75-openshift-default-75dc4675bf-qbhjq" Nov 03 02:22:34.357932 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:34.357904 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-ea5c75-openshift-default-75dc4675bf-qbhjq" Nov 03 02:22:34.402075 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:34.402039 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:22:34.792801 ip-10-0-136-16 kubenswrapper[5437]: W1103 02:22:34.792762 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38fbd6ae_90e3_4eac_aef8_8fa3ea9cb98c.slice/crio-959d1d8c861933ea687b09f230ad814311988afb892a9b2b75d3bc0fc7e91bc6 WatchSource:0}: Error finding container 959d1d8c861933ea687b09f230ad814311988afb892a9b2b75d3bc0fc7e91bc6: Status 404 returned error can't find the container with id 959d1d8c861933ea687b09f230ad814311988afb892a9b2b75d3bc0fc7e91bc6 Nov 03 02:22:34.795797 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:34.795752 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 02:22:34.795891 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:34.795847 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 02:22:34.795932 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:34.795891 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 02:22:34.796258 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:34.796235 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-ea5c75-openshift-default-75dc4675bf-qbhjq"] Nov 03 02:22:35.365991 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:35.365956 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:22:35.497770 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:35.497735 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ea5c75-openshift-default-75dc4675bf-qbhjq" event={"ID":"38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c","Type":"ContainerStarted","Data":"79d094231d716860f9c35455796545cebee39a73796351fa8f0816d924774e07"} Nov 03 02:22:35.497953 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:35.497791 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ea5c75-openshift-default-75dc4675bf-qbhjq" event={"ID":"38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c","Type":"ContainerStarted","Data":"959d1d8c861933ea687b09f230ad814311988afb892a9b2b75d3bc0fc7e91bc6"} Nov 03 02:22:35.616515 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:35.616429 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:22:35.674737 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:35.674683 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-ea5c75-openshift-default-75dc4675bf-qbhjq" podStartSLOduration=2.674669363 podStartE2EDuration="2.674669363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:22:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:22:35.67268169 +0000 UTC m=+3837.702163984" watchObservedRunningTime="2026-11-03 02:22:35.674669363 +0000 UTC m=+3837.704151658" Nov 03 02:22:36.130179 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:36.130129 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:22:36.358234 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:36.358207 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-ea5c75-openshift-default-75dc4675bf-qbhjq" Nov 03 02:22:36.365235 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:36.365210 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-ea5c75-openshift-default-75dc4675bf-qbhjq" Nov 03 02:22:36.507075 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:36.507044 5437 generic.go:356] "Generic (PLEG): container finished" podID="6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea" containerID="69f899906e8c170453bf9a7a42d26165c8df0a3ac53a5a98b28acb4d21b7cd47" exitCode=0 Nov 03 02:22:36.507568 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:36.507094 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-50ca6d-openshift-default-6b7b98d8c7-499js" event={"ID":"6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea","Type":"ContainerDied","Data":"69f899906e8c170453bf9a7a42d26165c8df0a3ac53a5a98b28acb4d21b7cd47"} Nov 03 02:22:36.507568 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:36.507167 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-50ca6d-openshift-default-6b7b98d8c7-499js" event={"ID":"6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea","Type":"ContainerDied","Data":"ab5acd4584d51e3da66baa13d78435cd5e834185015e5a7038e364f7139ed420"} Nov 03 02:22:36.507568 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:36.507195 5437 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ab5acd4584d51e3da66baa13d78435cd5e834185015e5a7038e364f7139ed420" Nov 03 02:22:36.507568 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:36.507217 5437 scope.go:122] "RemoveContainer" containerID="659c9112015d4f3ab0715fd52664d159b8a66d7dc8ba8c91c0ff6a85772096b4" Nov 03 02:22:36.507568 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:36.507311 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-ea5c75-openshift-default-75dc4675bf-qbhjq" Nov 03 02:22:36.508763 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:36.508744 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-ea5c75-openshift-default-75dc4675bf-qbhjq" Nov 03 02:22:36.514174 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:36.514156 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-50ca6d-openshift-default-6b7b98d8c7-499js" Nov 03 02:22:36.604020 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:36.603988 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea-istio-token\" (UniqueName: \"kubernetes.io/projected/6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea-istio-token\") pod \"6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea\" (UID: \"6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea\") " Nov 03 02:22:36.604214 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:36.604052 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea-workload-certs\") pod \"6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea\" (UID: \"6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea\") " Nov 03 02:22:36.604214 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:36.604081 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea-istiod-ca-cert\") pod \"6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea\" (UID: \"6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea\") " Nov 03 02:22:36.604214 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:36.604128 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea-kube-api-access-bnmb4\" (UniqueName: \"kubernetes.io/projected/6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea-kube-api-access-bnmb4\") pod \"6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea\" (UID: \"6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea\") " Nov 03 02:22:36.604214 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:36.604176 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea-istio-podinfo\") pod \"6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea\" (UID: \"6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea\") " Nov 03 02:22:36.604214 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:36.604212 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea-istio-data\") pod \"6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea\" (UID: \"6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea\") " Nov 03 02:22:36.604517 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:36.604361 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea-workload-certs" pod "6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea" (UID: "6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:22:36.604517 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:36.604400 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea-credential-socket\") pod \"6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea\" (UID: \"6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea\") " Nov 03 02:22:36.604517 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:36.604441 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea-workload-socket\") pod \"6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea\" (UID: \"6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea\") " Nov 03 02:22:36.604517 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:36.604489 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea-istio-envoy\") pod \"6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea\" (UID: \"6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea\") " Nov 03 02:22:36.604733 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:36.604653 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea-credential-socket" pod "6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea" (UID: "6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:22:36.604733 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:36.604579 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea-istiod-ca-cert" pod "6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea" (UID: "6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:22:36.604733 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:36.604671 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea-istio-data" pod "6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea" (UID: "6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:22:36.604863 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:36.604764 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea-workload-socket" pod "6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea" (UID: "6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:22:36.605746 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:36.605538 5437 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea-credential-socket\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:22:36.605746 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:36.605566 5437 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea-workload-socket\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:22:36.605746 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:36.605582 5437 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea-workload-certs\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:22:36.605746 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:36.605662 5437 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea-istiod-ca-cert\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:22:36.605746 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:36.605679 5437 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea-istio-data\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:22:36.606597 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:36.606575 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea-istio-token" pod "6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea" (UID: "6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:22:36.606597 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:36.606589 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea-kube-api-access-bnmb4" pod "6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea" (UID: "6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea"). InnerVolumeSpecName "kube-api-access-bnmb4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:22:36.606781 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:36.606654 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea-istio-podinfo" pod "6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea" (UID: "6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:22:36.606847 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:36.606831 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea-istio-envoy" pod "6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea" (UID: "6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:22:36.707000 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:36.706920 5437 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea-istio-envoy\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:22:36.707000 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:36.706949 5437 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea-istio-token\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:22:36.707000 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:36.706959 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnmb4\" (UniqueName: \"kubernetes.io/projected/6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea-kube-api-access-bnmb4\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:22:36.707000 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:36.706968 5437 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea-istio-podinfo\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:22:37.407123 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:37.407083 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:22:37.515108 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:37.515075 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-50ca6d-openshift-default-6b7b98d8c7-499js" Nov 03 02:22:37.660567 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:37.660465 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:22:37.806746 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:37.806712 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-50ca6d-openshift-default-6b7b98d8c7-499js"] Nov 03 02:22:37.814996 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:37.814953 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-50ca6d-openshift-default-6b7b98d8c7-499js"] Nov 03 02:22:38.089902 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:38.089873 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea" path="/var/lib/kubelet/pods/6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea/volumes" Nov 03 02:22:38.517825 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:38.517730 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:22:38.778743 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:38.778649 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:22:39.100711 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:39.100673 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:22:40.507213 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:40.507181 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-d8cc55-6f4d7f5cf4-njhfg"] Nov 03 02:22:40.507728 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:40.507402 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d8cc55-6f4d7f5cf4-njhfg" podUID="8721e215-270a-4092-b0ef-16fb74a4b863" containerName="maas-api" containerID="cri-o://a92afd386cf5ba83a8293f695292b3745830db6278b12fc1f8377b2e4e1e1ee5" gracePeriod=30 Nov 03 02:22:40.658799 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:40.658760 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:22:40.785775 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:40.785751 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d8cc55-6f4d7f5cf4-njhfg" Nov 03 02:22:40.961515 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:40.961481 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8721e215-270a-4092-b0ef-16fb74a4b863-kube-api-access-brdzj\" (UniqueName: \"kubernetes.io/projected/8721e215-270a-4092-b0ef-16fb74a4b863-kube-api-access-brdzj\") pod \"8721e215-270a-4092-b0ef-16fb74a4b863\" (UID: \"8721e215-270a-4092-b0ef-16fb74a4b863\") " Nov 03 02:22:40.961699 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:40.961522 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8721e215-270a-4092-b0ef-16fb74a4b863-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8721e215-270a-4092-b0ef-16fb74a4b863-metrics-certs\") pod \"8721e215-270a-4092-b0ef-16fb74a4b863\" (UID: \"8721e215-270a-4092-b0ef-16fb74a4b863\") " Nov 03 02:22:40.961699 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:40.961661 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8721e215-270a-4092-b0ef-16fb74a4b863-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8721e215-270a-4092-b0ef-16fb74a4b863-maas-api-tls\") pod \"8721e215-270a-4092-b0ef-16fb74a4b863\" (UID: \"8721e215-270a-4092-b0ef-16fb74a4b863\") " Nov 03 02:22:40.961789 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:40.961716 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8721e215-270a-4092-b0ef-16fb74a4b863-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8721e215-270a-4092-b0ef-16fb74a4b863-openshift-service-ca\") pod \"8721e215-270a-4092-b0ef-16fb74a4b863\" (UID: \"8721e215-270a-4092-b0ef-16fb74a4b863\") " Nov 03 02:22:40.962126 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:40.962096 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8721e215-270a-4092-b0ef-16fb74a4b863-openshift-service-ca" pod "8721e215-270a-4092-b0ef-16fb74a4b863" (UID: "8721e215-270a-4092-b0ef-16fb74a4b863"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:22:40.963763 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:40.963733 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8721e215-270a-4092-b0ef-16fb74a4b863-kube-api-access-brdzj" pod "8721e215-270a-4092-b0ef-16fb74a4b863" (UID: "8721e215-270a-4092-b0ef-16fb74a4b863"). InnerVolumeSpecName "kube-api-access-brdzj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:22:40.963869 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:40.963761 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8721e215-270a-4092-b0ef-16fb74a4b863-maas-api-tls" pod "8721e215-270a-4092-b0ef-16fb74a4b863" (UID: "8721e215-270a-4092-b0ef-16fb74a4b863"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:22:40.963869 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:40.963812 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8721e215-270a-4092-b0ef-16fb74a4b863-metrics-certs" pod "8721e215-270a-4092-b0ef-16fb74a4b863" (UID: "8721e215-270a-4092-b0ef-16fb74a4b863"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:22:41.063047 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:41.063005 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brdzj\" (UniqueName: \"kubernetes.io/projected/8721e215-270a-4092-b0ef-16fb74a4b863-kube-api-access-brdzj\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:22:41.063047 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:41.063040 5437 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8721e215-270a-4092-b0ef-16fb74a4b863-metrics-certs\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:22:41.063047 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:41.063054 5437 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8721e215-270a-4092-b0ef-16fb74a4b863-maas-api-tls\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:22:41.063306 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:41.063067 5437 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8721e215-270a-4092-b0ef-16fb74a4b863-openshift-service-ca\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:22:41.539956 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:41.539864 5437 generic.go:356] "Generic (PLEG): container finished" podID="8721e215-270a-4092-b0ef-16fb74a4b863" containerID="a92afd386cf5ba83a8293f695292b3745830db6278b12fc1f8377b2e4e1e1ee5" exitCode=0 Nov 03 02:22:41.539956 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:41.539912 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d8cc55-6f4d7f5cf4-njhfg" Nov 03 02:22:41.539956 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:41.539940 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d8cc55-6f4d7f5cf4-njhfg" event={"ID":"8721e215-270a-4092-b0ef-16fb74a4b863","Type":"ContainerDied","Data":"a92afd386cf5ba83a8293f695292b3745830db6278b12fc1f8377b2e4e1e1ee5"} Nov 03 02:22:41.540493 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:41.539998 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d8cc55-6f4d7f5cf4-njhfg" event={"ID":"8721e215-270a-4092-b0ef-16fb74a4b863","Type":"ContainerDied","Data":"c1b3856842e2659599ba1b775caa7379d7c3d6deb20dd46164fa571936efc9d9"} Nov 03 02:22:41.540493 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:41.540021 5437 scope.go:122] "RemoveContainer" containerID="a92afd386cf5ba83a8293f695292b3745830db6278b12fc1f8377b2e4e1e1ee5" Nov 03 02:22:41.550341 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:41.550322 5437 scope.go:122] "RemoveContainer" containerID="a92afd386cf5ba83a8293f695292b3745830db6278b12fc1f8377b2e4e1e1ee5" Nov 03 02:22:41.550669 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:22:41.550640 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a92afd386cf5ba83a8293f695292b3745830db6278b12fc1f8377b2e4e1e1ee5\": container with ID starting with a92afd386cf5ba83a8293f695292b3745830db6278b12fc1f8377b2e4e1e1ee5 not found: ID does not exist" containerID="a92afd386cf5ba83a8293f695292b3745830db6278b12fc1f8377b2e4e1e1ee5" Nov 03 02:22:41.550724 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:41.550670 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a92afd386cf5ba83a8293f695292b3745830db6278b12fc1f8377b2e4e1e1ee5"} err="failed to get container status \"a92afd386cf5ba83a8293f695292b3745830db6278b12fc1f8377b2e4e1e1ee5\": rpc error: code = NotFound desc = could not find container \"a92afd386cf5ba83a8293f695292b3745830db6278b12fc1f8377b2e4e1e1ee5\": container with ID starting with a92afd386cf5ba83a8293f695292b3745830db6278b12fc1f8377b2e4e1e1ee5 not found: ID does not exist" Nov 03 02:22:41.632767 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:41.632727 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:22:41.710754 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:41.710720 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-d8cc55-6f4d7f5cf4-njhfg"] Nov 03 02:22:41.762310 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:41.762274 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-d8cc55-6f4d7f5cf4-njhfg"] Nov 03 02:22:41.849944 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:41.849903 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:22:41.974726 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:41.974689 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-ea5c75-67b8d7c68c-vv2np"] Nov 03 02:22:41.975288 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:41.975274 5437 cpu_manager.go:414] "containerMap: removing container" podUID="6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea" containerName="istio-proxy" Nov 03 02:22:41.975336 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:41.975289 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea" containerName="istio-proxy" Nov 03 02:22:41.975392 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:41.975337 5437 cpu_manager.go:414] "containerMap: removing container" podUID="8721e215-270a-4092-b0ef-16fb74a4b863" containerName="maas-api" Nov 03 02:22:41.975392 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:41.975343 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8721e215-270a-4092-b0ef-16fb74a4b863" containerName="maas-api" Nov 03 02:22:41.975512 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:41.975459 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea" containerName="istio-proxy" Nov 03 02:22:41.975512 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:41.975480 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="8721e215-270a-4092-b0ef-16fb74a4b863" containerName="maas-api" Nov 03 02:22:41.978729 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:41.978713 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ea5c75-67b8d7c68c-vv2np" Nov 03 02:22:42.032598 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:42.032567 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-ea5c75-67b8d7c68c-vv2np"] Nov 03 02:22:42.040652 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:42.040624 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-ea5c75\"" Nov 03 02:22:42.062032 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:42.062001 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:22:42.081248 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:42.081211 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a753f6ce-4198-4196-96de-73ff6c81b32a-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-ea5c75-67b8d7c68c-vv2np\" (UID: \"a753f6ce-4198-4196-96de-73ff6c81b32a\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ea5c75-67b8d7c68c-vv2np" Nov 03 02:22:42.081428 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:42.081291 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5ncz\" (UniqueName: \"kubernetes.io/projected/a753f6ce-4198-4196-96de-73ff6c81b32a-kube-api-access-k5ncz\") pod \"payload-pre-processing-e2e-extmod-a-ea5c75-67b8d7c68c-vv2np\" (UID: \"a753f6ce-4198-4196-96de-73ff6c81b32a\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ea5c75-67b8d7c68c-vv2np" Nov 03 02:22:42.097323 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:42.096876 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8721e215-270a-4092-b0ef-16fb74a4b863" path="/var/lib/kubelet/pods/8721e215-270a-4092-b0ef-16fb74a4b863/volumes" Nov 03 02:22:42.108433 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:42.108336 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-ea5c75-dockercfg-xv5gs\"" Nov 03 02:22:42.137598 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:42.137567 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-ea5c75-569b6776c9-pqphg"] Nov 03 02:22:42.138167 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:42.138154 5437 cpu_manager.go:414] "containerMap: removing container" podUID="6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea" containerName="istio-proxy" Nov 03 02:22:42.138212 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:42.138169 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea" containerName="istio-proxy" Nov 03 02:22:42.138348 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:42.138336 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="6cc0aaa6-f284-44a2-a2c8-bb0283ce68ea" containerName="istio-proxy" Nov 03 02:22:42.145091 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:42.145059 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-ea5c75-569b6776c9-pqphg" Nov 03 02:22:42.182357 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:42.182318 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a753f6ce-4198-4196-96de-73ff6c81b32a-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-ea5c75-67b8d7c68c-vv2np\" (UID: \"a753f6ce-4198-4196-96de-73ff6c81b32a\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ea5c75-67b8d7c68c-vv2np" Nov 03 02:22:42.182552 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:42.182419 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5ncz\" (UniqueName: \"kubernetes.io/projected/a753f6ce-4198-4196-96de-73ff6c81b32a-kube-api-access-k5ncz\") pod \"payload-pre-processing-e2e-extmod-a-ea5c75-67b8d7c68c-vv2np\" (UID: \"a753f6ce-4198-4196-96de-73ff6c81b32a\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ea5c75-67b8d7c68c-vv2np" Nov 03 02:22:42.183189 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:42.183164 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a753f6ce-4198-4196-96de-73ff6c81b32a-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-ea5c75-67b8d7c68c-vv2np\" (UID: \"a753f6ce-4198-4196-96de-73ff6c81b32a\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ea5c75-67b8d7c68c-vv2np" Nov 03 02:22:42.198470 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:42.198442 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-ea5c75-569b6776c9-pqphg"] Nov 03 02:22:42.245210 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:42.245174 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5ncz\" (UniqueName: \"kubernetes.io/projected/a753f6ce-4198-4196-96de-73ff6c81b32a-kube-api-access-k5ncz\") pod \"payload-pre-processing-e2e-extmod-a-ea5c75-67b8d7c68c-vv2np\" (UID: \"a753f6ce-4198-4196-96de-73ff6c81b32a\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ea5c75-67b8d7c68c-vv2np" Nov 03 02:22:42.284449 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:42.284416 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85d7dce8-f705-45e7-a5a9-6a5ff7ef1f2d-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-ea5c75-569b6776c9-pqphg\" (UID: \"85d7dce8-f705-45e7-a5a9-6a5ff7ef1f2d\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-ea5c75-569b6776c9-pqphg" Nov 03 02:22:42.284607 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:42.284493 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6m6p4\" (UniqueName: \"kubernetes.io/projected/85d7dce8-f705-45e7-a5a9-6a5ff7ef1f2d-kube-api-access-6m6p4\") pod \"payload-processing-e2e-extmod-a-ea5c75-569b6776c9-pqphg\" (UID: \"85d7dce8-f705-45e7-a5a9-6a5ff7ef1f2d\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-ea5c75-569b6776c9-pqphg" Nov 03 02:22:42.293129 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:42.293095 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ea5c75-67b8d7c68c-vv2np" Nov 03 02:22:42.385401 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:42.385283 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85d7dce8-f705-45e7-a5a9-6a5ff7ef1f2d-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-ea5c75-569b6776c9-pqphg\" (UID: \"85d7dce8-f705-45e7-a5a9-6a5ff7ef1f2d\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-ea5c75-569b6776c9-pqphg" Nov 03 02:22:42.385401 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:42.385346 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6m6p4\" (UniqueName: \"kubernetes.io/projected/85d7dce8-f705-45e7-a5a9-6a5ff7ef1f2d-kube-api-access-6m6p4\") pod \"payload-processing-e2e-extmod-a-ea5c75-569b6776c9-pqphg\" (UID: \"85d7dce8-f705-45e7-a5a9-6a5ff7ef1f2d\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-ea5c75-569b6776c9-pqphg" Nov 03 02:22:42.386141 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:42.386110 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85d7dce8-f705-45e7-a5a9-6a5ff7ef1f2d-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-ea5c75-569b6776c9-pqphg\" (UID: \"85d7dce8-f705-45e7-a5a9-6a5ff7ef1f2d\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-ea5c75-569b6776c9-pqphg" Nov 03 02:22:42.448178 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:42.448144 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6m6p4\" (UniqueName: \"kubernetes.io/projected/85d7dce8-f705-45e7-a5a9-6a5ff7ef1f2d-kube-api-access-6m6p4\") pod \"payload-processing-e2e-extmod-a-ea5c75-569b6776c9-pqphg\" (UID: \"85d7dce8-f705-45e7-a5a9-6a5ff7ef1f2d\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-ea5c75-569b6776c9-pqphg" Nov 03 02:22:42.463747 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:42.463719 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-ea5c75-569b6776c9-pqphg" Nov 03 02:22:42.464938 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:42.464901 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:22:42.667636 ip-10-0-136-16 kubenswrapper[5437]: W1103 02:22:42.667607 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda753f6ce_4198_4196_96de_73ff6c81b32a.slice/crio-72fc3a6f25a8706cae6e1a3b3e7efdcbeaf4fda22b6e525e477ea032f8839808 WatchSource:0}: Error finding container 72fc3a6f25a8706cae6e1a3b3e7efdcbeaf4fda22b6e525e477ea032f8839808: Status 404 returned error can't find the container with id 72fc3a6f25a8706cae6e1a3b3e7efdcbeaf4fda22b6e525e477ea032f8839808 Nov 03 02:22:42.671874 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:42.671848 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-ea5c75-67b8d7c68c-vv2np"] Nov 03 02:22:42.718029 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:42.717985 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:22:42.834827 ip-10-0-136-16 kubenswrapper[5437]: W1103 02:22:42.834800 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85d7dce8_f705_45e7_a5a9_6a5ff7ef1f2d.slice/crio-13d43a766d85587810fbb5b846c13403da3938043e357f6878a5e914f2ba58e9 WatchSource:0}: Error finding container 13d43a766d85587810fbb5b846c13403da3938043e357f6878a5e914f2ba58e9: Status 404 returned error can't find the container with id 13d43a766d85587810fbb5b846c13403da3938043e357f6878a5e914f2ba58e9 Nov 03 02:22:42.837553 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:42.837528 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-ea5c75-569b6776c9-pqphg"] Nov 03 02:22:43.083102 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:43.083066 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:22:43.556257 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:43.556211 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ea5c75-67b8d7c68c-vv2np" event={"ID":"a753f6ce-4198-4196-96de-73ff6c81b32a","Type":"ContainerStarted","Data":"b2e0c7216f1cb0690209780cdc138eb315b59a7d1da4f1d0d73040441c65c2e6"} Nov 03 02:22:43.556504 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:43.556277 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ea5c75-67b8d7c68c-vv2np" event={"ID":"a753f6ce-4198-4196-96de-73ff6c81b32a","Type":"ContainerStarted","Data":"72fc3a6f25a8706cae6e1a3b3e7efdcbeaf4fda22b6e525e477ea032f8839808"} Nov 03 02:22:43.556504 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:43.556306 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ea5c75-67b8d7c68c-vv2np" Nov 03 02:22:43.557921 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:43.557891 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-ea5c75-569b6776c9-pqphg" event={"ID":"85d7dce8-f705-45e7-a5a9-6a5ff7ef1f2d","Type":"ContainerStarted","Data":"b66ea622c395d947328eeb56fc21088876da60cf7536b48f87e8b36a30b36881"} Nov 03 02:22:43.558022 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:43.557942 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-ea5c75-569b6776c9-pqphg" Nov 03 02:22:43.558022 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:43.557960 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-ea5c75-569b6776c9-pqphg" event={"ID":"85d7dce8-f705-45e7-a5a9-6a5ff7ef1f2d","Type":"ContainerStarted","Data":"13d43a766d85587810fbb5b846c13403da3938043e357f6878a5e914f2ba58e9"} Nov 03 02:22:43.677593 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:43.677525 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ea5c75-67b8d7c68c-vv2np" podStartSLOduration=2.6775067740000003 podStartE2EDuration="2.677506774s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:22:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:22:43.677052014 +0000 UTC m=+3845.706534310" watchObservedRunningTime="2026-11-03 02:22:43.677506774 +0000 UTC m=+3845.706989069" Nov 03 02:22:43.798541 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:43.798474 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-ea5c75-569b6776c9-pqphg" podStartSLOduration=1.798454248 podStartE2EDuration="1.798454248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:22:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:22:43.796210382 +0000 UTC m=+3845.825692680" watchObservedRunningTime="2026-11-03 02:22:43.798454248 +0000 UTC m=+3845.827936546" Nov 03 02:22:44.034260 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:22:44.034171 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 02:22:46.780930 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:46.780889 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:22:47.032895 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:47.032819 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:22:47.573187 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:47.573150 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:22:50.644732 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:50.644696 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:22:51.114380 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:51.114338 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-7ff5cb8dc-82dcd"] Nov 03 02:22:51.118753 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:51.118730 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7ff5cb8dc-82dcd" Nov 03 02:22:51.172876 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:51.172841 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7ff5cb8dc-82dcd"] Nov 03 02:22:51.188134 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:51.188096 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11c05ff6-f766-450d-87f0-a83013d3183c-plugins-config-volume\") pod \"payload-processing-external-model-7ff5cb8dc-82dcd\" (UID: \"11c05ff6-f766-450d-87f0-a83013d3183c\") " pod="models-as-a-service/payload-processing-external-model-7ff5cb8dc-82dcd" Nov 03 02:22:51.188339 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:51.188241 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/11c05ff6-f766-450d-87f0-a83013d3183c-routing-overlay-volume\") pod \"payload-processing-external-model-7ff5cb8dc-82dcd\" (UID: \"11c05ff6-f766-450d-87f0-a83013d3183c\") " pod="models-as-a-service/payload-processing-external-model-7ff5cb8dc-82dcd" Nov 03 02:22:51.188339 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:51.188272 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/11c05ff6-f766-450d-87f0-a83013d3183c-provider-credentials\") pod \"payload-processing-external-model-7ff5cb8dc-82dcd\" (UID: \"11c05ff6-f766-450d-87f0-a83013d3183c\") " pod="models-as-a-service/payload-processing-external-model-7ff5cb8dc-82dcd" Nov 03 02:22:51.188493 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:51.188473 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvjhh\" (UniqueName: \"kubernetes.io/projected/11c05ff6-f766-450d-87f0-a83013d3183c-kube-api-access-wvjhh\") pod \"payload-processing-external-model-7ff5cb8dc-82dcd\" (UID: \"11c05ff6-f766-450d-87f0-a83013d3183c\") " pod="models-as-a-service/payload-processing-external-model-7ff5cb8dc-82dcd" Nov 03 02:22:51.221816 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:51.221775 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Nov 03 02:22:51.221816 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:51.221772 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Nov 03 02:22:51.222053 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:51.221885 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Nov 03 02:22:51.222053 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:51.221917 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-4fa7cdd724cd\"" Nov 03 02:22:51.222053 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:51.221930 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-t2v42\"" Nov 03 02:22:51.224633 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:51.224619 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Nov 03 02:22:51.289296 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:51.289266 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvjhh\" (UniqueName: \"kubernetes.io/projected/11c05ff6-f766-450d-87f0-a83013d3183c-kube-api-access-wvjhh\") pod \"payload-processing-external-model-7ff5cb8dc-82dcd\" (UID: \"11c05ff6-f766-450d-87f0-a83013d3183c\") " pod="models-as-a-service/payload-processing-external-model-7ff5cb8dc-82dcd" Nov 03 02:22:51.289475 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:51.289348 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11c05ff6-f766-450d-87f0-a83013d3183c-plugins-config-volume\") pod \"payload-processing-external-model-7ff5cb8dc-82dcd\" (UID: \"11c05ff6-f766-450d-87f0-a83013d3183c\") " pod="models-as-a-service/payload-processing-external-model-7ff5cb8dc-82dcd" Nov 03 02:22:51.289475 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:51.289456 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/11c05ff6-f766-450d-87f0-a83013d3183c-routing-overlay-volume\") pod \"payload-processing-external-model-7ff5cb8dc-82dcd\" (UID: \"11c05ff6-f766-450d-87f0-a83013d3183c\") " pod="models-as-a-service/payload-processing-external-model-7ff5cb8dc-82dcd" Nov 03 02:22:51.289560 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:51.289478 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/11c05ff6-f766-450d-87f0-a83013d3183c-provider-credentials\") pod \"payload-processing-external-model-7ff5cb8dc-82dcd\" (UID: \"11c05ff6-f766-450d-87f0-a83013d3183c\") " pod="models-as-a-service/payload-processing-external-model-7ff5cb8dc-82dcd" Nov 03 02:22:51.290093 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:51.290071 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/11c05ff6-f766-450d-87f0-a83013d3183c-routing-overlay-volume\") pod \"payload-processing-external-model-7ff5cb8dc-82dcd\" (UID: \"11c05ff6-f766-450d-87f0-a83013d3183c\") " pod="models-as-a-service/payload-processing-external-model-7ff5cb8dc-82dcd" Nov 03 02:22:51.290216 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:51.290192 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11c05ff6-f766-450d-87f0-a83013d3183c-plugins-config-volume\") pod \"payload-processing-external-model-7ff5cb8dc-82dcd\" (UID: \"11c05ff6-f766-450d-87f0-a83013d3183c\") " pod="models-as-a-service/payload-processing-external-model-7ff5cb8dc-82dcd" Nov 03 02:22:51.291985 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:51.291965 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/11c05ff6-f766-450d-87f0-a83013d3183c-provider-credentials\") pod \"payload-processing-external-model-7ff5cb8dc-82dcd\" (UID: \"11c05ff6-f766-450d-87f0-a83013d3183c\") " pod="models-as-a-service/payload-processing-external-model-7ff5cb8dc-82dcd" Nov 03 02:22:51.353183 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:51.353149 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvjhh\" (UniqueName: \"kubernetes.io/projected/11c05ff6-f766-450d-87f0-a83013d3183c-kube-api-access-wvjhh\") pod \"payload-processing-external-model-7ff5cb8dc-82dcd\" (UID: \"11c05ff6-f766-450d-87f0-a83013d3183c\") " pod="models-as-a-service/payload-processing-external-model-7ff5cb8dc-82dcd" Nov 03 02:22:51.433472 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:51.433389 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7ff5cb8dc-82dcd" Nov 03 02:22:51.821051 ip-10-0-136-16 kubenswrapper[5437]: W1103 02:22:51.821003 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11c05ff6_f766_450d_87f0_a83013d3183c.slice/crio-b7739f7e01617192951f49630eaa9fb06d1768fac36032ca553c4476d6b31683 WatchSource:0}: Error finding container b7739f7e01617192951f49630eaa9fb06d1768fac36032ca553c4476d6b31683: Status 404 returned error can't find the container with id b7739f7e01617192951f49630eaa9fb06d1768fac36032ca553c4476d6b31683 Nov 03 02:22:51.823449 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:51.823318 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7ff5cb8dc-82dcd"] Nov 03 02:22:52.473583 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:52.473546 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:22:52.624680 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:52.624641 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7ff5cb8dc-82dcd" event={"ID":"11c05ff6-f766-450d-87f0-a83013d3183c","Type":"ContainerStarted","Data":"8148ab73c172214f2794fbd2adfeb5d119f563580918059329b917dbb9e51391"} Nov 03 02:22:52.624883 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:52.624693 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7ff5cb8dc-82dcd" event={"ID":"11c05ff6-f766-450d-87f0-a83013d3183c","Type":"ContainerStarted","Data":"b7739f7e01617192951f49630eaa9fb06d1768fac36032ca553c4476d6b31683"} Nov 03 02:22:52.624883 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:52.624719 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-7ff5cb8dc-82dcd" Nov 03 02:22:52.746551 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:52.746446 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-7ff5cb8dc-82dcd" podStartSLOduration=1.746426011 podStartE2EDuration="1.746426011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:22:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:22:52.744806515 +0000 UTC m=+3854.774288809" watchObservedRunningTime="2026-11-03 02:22:52.746426011 +0000 UTC m=+3854.775908307" Nov 03 02:22:54.576990 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:54.576956 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ea5c75-67b8d7c68c-vv2np" Nov 03 02:22:54.577471 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:54.577028 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-ea5c75-569b6776c9-pqphg" Nov 03 02:22:57.274904 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:57.274865 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:22:57.737925 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:22:57.737886 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:23:02.079914 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:02.079864 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:23:02.605099 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:02.605068 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:23:03.634972 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:03.634940 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-7ff5cb8dc-82dcd" Nov 03 02:23:06.840614 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:06.840580 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:23:11.530862 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:11.530813 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:23:14.034541 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:23:14.034498 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 02:23:16.286331 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:16.286292 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:23:21.120311 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:21.120276 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:23:25.396979 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:25.396941 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:23:26.498138 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:26.498084 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:23:27.496325 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:27.496275 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:23:27.780248 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:27.780163 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:23:29.032689 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:29.032649 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:23:29.393540 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:29.393500 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:23:29.645932 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:29.645837 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:23:29.852609 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:29.852573 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:23:30.365052 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:30.365016 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:23:30.676858 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:30.676755 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:23:31.133738 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:31.133707 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-tz9hh"] Nov 03 02:23:31.133930 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:31.133912 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-tz9hh" podUID="a5837de8-ead2-4681-8689-3d5fb3bd3ae1" containerName="istio-proxy" containerID="cri-o://5af73d02f93d7bd464fd88c19f774b1b96de6159e16946af43b5994cf03b863c" gracePeriod=30 Nov 03 02:23:31.199841 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:31.199803 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:23:31.994765 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:31.994732 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:23:32.127345 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:32.127315 5437 patch_prober.go:29] interesting pod/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-tz9hh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 02:23:32.127542 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:32.127395 5437 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-tz9hh" podUID="a5837de8-ead2-4681-8689-3d5fb3bd3ae1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 02:23:32.513272 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:32.513232 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:23:32.861702 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:32.861666 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:23:33.072236 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:33.072202 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:23:34.519946 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:34.519904 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:23:35.433395 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:35.433347 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:23:36.407578 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:36.407507 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-tz9hh" Nov 03 02:23:36.443071 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:36.443035 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:23:36.459016 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:36.458983 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a5837de8-ead2-4681-8689-3d5fb3bd3ae1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5837de8-ead2-4681-8689-3d5fb3bd3ae1-istio-podinfo\") pod \"a5837de8-ead2-4681-8689-3d5fb3bd3ae1\" (UID: \"a5837de8-ead2-4681-8689-3d5fb3bd3ae1\") " Nov 03 02:23:36.459203 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:36.459078 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5837de8-ead2-4681-8689-3d5fb3bd3ae1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5837de8-ead2-4681-8689-3d5fb3bd3ae1-workload-socket\") pod \"a5837de8-ead2-4681-8689-3d5fb3bd3ae1\" (UID: \"a5837de8-ead2-4681-8689-3d5fb3bd3ae1\") " Nov 03 02:23:36.459203 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:36.459151 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5837de8-ead2-4681-8689-3d5fb3bd3ae1-istio-token\" (UniqueName: \"kubernetes.io/projected/a5837de8-ead2-4681-8689-3d5fb3bd3ae1-istio-token\") pod \"a5837de8-ead2-4681-8689-3d5fb3bd3ae1\" (UID: \"a5837de8-ead2-4681-8689-3d5fb3bd3ae1\") " Nov 03 02:23:36.459358 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:36.459336 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5837de8-ead2-4681-8689-3d5fb3bd3ae1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5837de8-ead2-4681-8689-3d5fb3bd3ae1-istio-data\") pod \"a5837de8-ead2-4681-8689-3d5fb3bd3ae1\" (UID: \"a5837de8-ead2-4681-8689-3d5fb3bd3ae1\") " Nov 03 02:23:36.459465 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:36.459400 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5837de8-ead2-4681-8689-3d5fb3bd3ae1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5837de8-ead2-4681-8689-3d5fb3bd3ae1-workload-certs\") pod \"a5837de8-ead2-4681-8689-3d5fb3bd3ae1\" (UID: \"a5837de8-ead2-4681-8689-3d5fb3bd3ae1\") " Nov 03 02:23:36.459465 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:36.459434 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5837de8-ead2-4681-8689-3d5fb3bd3ae1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5837de8-ead2-4681-8689-3d5fb3bd3ae1-credential-socket\") pod \"a5837de8-ead2-4681-8689-3d5fb3bd3ae1\" (UID: \"a5837de8-ead2-4681-8689-3d5fb3bd3ae1\") " Nov 03 02:23:36.459465 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:36.459428 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5837de8-ead2-4681-8689-3d5fb3bd3ae1-workload-socket" pod "a5837de8-ead2-4681-8689-3d5fb3bd3ae1" (UID: "a5837de8-ead2-4681-8689-3d5fb3bd3ae1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:23:36.459647 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:36.459548 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5837de8-ead2-4681-8689-3d5fb3bd3ae1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5837de8-ead2-4681-8689-3d5fb3bd3ae1-istiod-ca-cert\") pod \"a5837de8-ead2-4681-8689-3d5fb3bd3ae1\" (UID: \"a5837de8-ead2-4681-8689-3d5fb3bd3ae1\") " Nov 03 02:23:36.459707 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:36.459668 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5837de8-ead2-4681-8689-3d5fb3bd3ae1-workload-certs" pod "a5837de8-ead2-4681-8689-3d5fb3bd3ae1" (UID: "a5837de8-ead2-4681-8689-3d5fb3bd3ae1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:23:36.459707 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:36.459677 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5837de8-ead2-4681-8689-3d5fb3bd3ae1-kube-api-access-hqcmp\" (UniqueName: \"kubernetes.io/projected/a5837de8-ead2-4681-8689-3d5fb3bd3ae1-kube-api-access-hqcmp\") pod \"a5837de8-ead2-4681-8689-3d5fb3bd3ae1\" (UID: \"a5837de8-ead2-4681-8689-3d5fb3bd3ae1\") " Nov 03 02:23:36.459809 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:36.459756 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5837de8-ead2-4681-8689-3d5fb3bd3ae1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5837de8-ead2-4681-8689-3d5fb3bd3ae1-istio-envoy\") pod \"a5837de8-ead2-4681-8689-3d5fb3bd3ae1\" (UID: \"a5837de8-ead2-4681-8689-3d5fb3bd3ae1\") " Nov 03 02:23:36.460140 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:36.460051 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5837de8-ead2-4681-8689-3d5fb3bd3ae1-istiod-ca-cert" pod "a5837de8-ead2-4681-8689-3d5fb3bd3ae1" (UID: "a5837de8-ead2-4681-8689-3d5fb3bd3ae1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:23:36.460256 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:36.460159 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5837de8-ead2-4681-8689-3d5fb3bd3ae1-credential-socket" pod "a5837de8-ead2-4681-8689-3d5fb3bd3ae1" (UID: "a5837de8-ead2-4681-8689-3d5fb3bd3ae1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:23:36.460522 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:36.460498 5437 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5837de8-ead2-4681-8689-3d5fb3bd3ae1-workload-socket\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:23:36.460708 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:36.460527 5437 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5837de8-ead2-4681-8689-3d5fb3bd3ae1-workload-certs\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:23:36.460708 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:36.460542 5437 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5837de8-ead2-4681-8689-3d5fb3bd3ae1-credential-socket\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:23:36.460708 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:36.460558 5437 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5837de8-ead2-4681-8689-3d5fb3bd3ae1-istiod-ca-cert\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:23:36.460975 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:36.460801 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5837de8-ead2-4681-8689-3d5fb3bd3ae1-istio-data" pod "a5837de8-ead2-4681-8689-3d5fb3bd3ae1" (UID: "a5837de8-ead2-4681-8689-3d5fb3bd3ae1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:23:36.461888 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:36.461859 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a5837de8-ead2-4681-8689-3d5fb3bd3ae1-istio-podinfo" pod "a5837de8-ead2-4681-8689-3d5fb3bd3ae1" (UID: "a5837de8-ead2-4681-8689-3d5fb3bd3ae1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:23:36.461994 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:36.461911 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5837de8-ead2-4681-8689-3d5fb3bd3ae1-istio-token" pod "a5837de8-ead2-4681-8689-3d5fb3bd3ae1" (UID: "a5837de8-ead2-4681-8689-3d5fb3bd3ae1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:23:36.462079 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:36.462055 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5837de8-ead2-4681-8689-3d5fb3bd3ae1-kube-api-access-hqcmp" pod "a5837de8-ead2-4681-8689-3d5fb3bd3ae1" (UID: "a5837de8-ead2-4681-8689-3d5fb3bd3ae1"). InnerVolumeSpecName "kube-api-access-hqcmp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:23:36.462404 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:36.462348 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5837de8-ead2-4681-8689-3d5fb3bd3ae1-istio-envoy" pod "a5837de8-ead2-4681-8689-3d5fb3bd3ae1" (UID: "a5837de8-ead2-4681-8689-3d5fb3bd3ae1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:23:36.561505 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:36.561471 5437 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5837de8-ead2-4681-8689-3d5fb3bd3ae1-istio-token\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:23:36.561505 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:36.561501 5437 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5837de8-ead2-4681-8689-3d5fb3bd3ae1-istio-data\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:23:36.561505 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:36.561513 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqcmp\" (UniqueName: \"kubernetes.io/projected/a5837de8-ead2-4681-8689-3d5fb3bd3ae1-kube-api-access-hqcmp\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:23:36.561759 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:36.561522 5437 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5837de8-ead2-4681-8689-3d5fb3bd3ae1-istio-envoy\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:23:36.561759 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:36.561531 5437 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5837de8-ead2-4681-8689-3d5fb3bd3ae1-istio-podinfo\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:23:36.701544 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:36.701454 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:23:36.913951 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:36.913910 5437 generic.go:356] "Generic (PLEG): container finished" podID="a5837de8-ead2-4681-8689-3d5fb3bd3ae1" containerID="5af73d02f93d7bd464fd88c19f774b1b96de6159e16946af43b5994cf03b863c" exitCode=0 Nov 03 02:23:36.914160 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:36.913960 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-tz9hh" Nov 03 02:23:36.914160 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:36.913973 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-tz9hh" event={"ID":"a5837de8-ead2-4681-8689-3d5fb3bd3ae1","Type":"ContainerDied","Data":"5af73d02f93d7bd464fd88c19f774b1b96de6159e16946af43b5994cf03b863c"} Nov 03 02:23:36.914160 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:36.914028 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-tz9hh" event={"ID":"a5837de8-ead2-4681-8689-3d5fb3bd3ae1","Type":"ContainerDied","Data":"759783293a6763e80ac67a64a3d49bee642a2da852123ddba5326024c3416542"} Nov 03 02:23:36.914160 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:36.914058 5437 scope.go:122] "RemoveContainer" containerID="5af73d02f93d7bd464fd88c19f774b1b96de6159e16946af43b5994cf03b863c" Nov 03 02:23:36.925587 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:36.925566 5437 scope.go:122] "RemoveContainer" containerID="5af73d02f93d7bd464fd88c19f774b1b96de6159e16946af43b5994cf03b863c" Nov 03 02:23:36.925906 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:23:36.925877 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5af73d02f93d7bd464fd88c19f774b1b96de6159e16946af43b5994cf03b863c\": container with ID starting with 5af73d02f93d7bd464fd88c19f774b1b96de6159e16946af43b5994cf03b863c not found: ID does not exist" containerID="5af73d02f93d7bd464fd88c19f774b1b96de6159e16946af43b5994cf03b863c" Nov 03 02:23:36.926009 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:36.925912 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5af73d02f93d7bd464fd88c19f774b1b96de6159e16946af43b5994cf03b863c"} err="failed to get container status \"5af73d02f93d7bd464fd88c19f774b1b96de6159e16946af43b5994cf03b863c\": rpc error: code = NotFound desc = could not find container \"5af73d02f93d7bd464fd88c19f774b1b96de6159e16946af43b5994cf03b863c\": container with ID starting with 5af73d02f93d7bd464fd88c19f774b1b96de6159e16946af43b5994cf03b863c not found: ID does not exist" Nov 03 02:23:37.084558 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:37.084516 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-tz9hh"] Nov 03 02:23:37.135743 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:37.135700 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-tz9hh"] Nov 03 02:23:37.281710 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:37.281668 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:23:38.088968 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:38.088935 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5837de8-ead2-4681-8689-3d5fb3bd3ae1" path="/var/lib/kubelet/pods/a5837de8-ead2-4681-8689-3d5fb3bd3ae1/volumes" Nov 03 02:23:38.248848 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:38.248814 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-3fb07e-7678cbbc5-kpm8s"] Nov 03 02:23:38.554762 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:38.554731 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:23:38.813183 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:38.813084 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:23:39.503131 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:39.503091 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:23:39.704067 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:39.704015 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:23:39.916892 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:39.916852 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:23:41.156892 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:41.156850 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:23:41.367924 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:41.367884 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:23:41.659115 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:41.659073 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:23:42.862332 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:42.862292 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:23:43.129711 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:43.129633 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7ff5cb8dc-82dcd"] Nov 03 02:23:43.195682 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:43.195642 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:23:43.249100 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:43.249056 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-3fb07e-7678cbbc5-kpm8s" podUID="db26a4c8-466a-4c1e-ba51-30bd6a076791" containerName="payload-processing" containerID="cri-o://b4c5a526f4da2db04711b6287428a2d9812139314c8f8f20342887f00fc1bba9" gracePeriod=25 Nov 03 02:23:43.416580 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:43.416487 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:23:43.545215 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:43.545181 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-3fb07e-7678cbbc5-kpm8s" Nov 03 02:23:43.616027 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:43.615996 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:23:43.652397 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:43.652337 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db26a4c8-466a-4c1e-ba51-30bd6a076791-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db26a4c8-466a-4c1e-ba51-30bd6a076791-plugins-config-volume\") pod \"db26a4c8-466a-4c1e-ba51-30bd6a076791\" (UID: \"db26a4c8-466a-4c1e-ba51-30bd6a076791\") " Nov 03 02:23:43.652583 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:43.652453 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db26a4c8-466a-4c1e-ba51-30bd6a076791-kube-api-access-8lmnf\" (UniqueName: \"kubernetes.io/projected/db26a4c8-466a-4c1e-ba51-30bd6a076791-kube-api-access-8lmnf\") pod \"db26a4c8-466a-4c1e-ba51-30bd6a076791\" (UID: \"db26a4c8-466a-4c1e-ba51-30bd6a076791\") " Nov 03 02:23:43.652739 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:43.652715 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db26a4c8-466a-4c1e-ba51-30bd6a076791-plugins-config-volume" pod "db26a4c8-466a-4c1e-ba51-30bd6a076791" (UID: "db26a4c8-466a-4c1e-ba51-30bd6a076791"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:23:43.654543 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:43.654517 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db26a4c8-466a-4c1e-ba51-30bd6a076791-kube-api-access-8lmnf" pod "db26a4c8-466a-4c1e-ba51-30bd6a076791" (UID: "db26a4c8-466a-4c1e-ba51-30bd6a076791"). InnerVolumeSpecName "kube-api-access-8lmnf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:23:43.754124 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:43.754035 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lmnf\" (UniqueName: \"kubernetes.io/projected/db26a4c8-466a-4c1e-ba51-30bd6a076791-kube-api-access-8lmnf\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:23:43.754124 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:43.754067 5437 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db26a4c8-466a-4c1e-ba51-30bd6a076791-plugins-config-volume\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:23:43.825903 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:43.825838 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:23:43.960574 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:43.960540 5437 generic.go:356] "Generic (PLEG): container finished" podID="db26a4c8-466a-4c1e-ba51-30bd6a076791" containerID="b4c5a526f4da2db04711b6287428a2d9812139314c8f8f20342887f00fc1bba9" exitCode=0 Nov 03 02:23:43.961055 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:43.960589 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-3fb07e-7678cbbc5-kpm8s" Nov 03 02:23:43.961055 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:43.960618 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-3fb07e-7678cbbc5-kpm8s" event={"ID":"db26a4c8-466a-4c1e-ba51-30bd6a076791","Type":"ContainerDied","Data":"b4c5a526f4da2db04711b6287428a2d9812139314c8f8f20342887f00fc1bba9"} Nov 03 02:23:43.961055 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:43.960674 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-3fb07e-7678cbbc5-kpm8s" event={"ID":"db26a4c8-466a-4c1e-ba51-30bd6a076791","Type":"ContainerDied","Data":"1939c741cd700ca8982ab83ded08319e45eabe8edd54ef32f8b99068e87bc921"} Nov 03 02:23:43.961055 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:43.960700 5437 scope.go:122] "RemoveContainer" containerID="b4c5a526f4da2db04711b6287428a2d9812139314c8f8f20342887f00fc1bba9" Nov 03 02:23:43.971001 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:43.970982 5437 scope.go:122] "RemoveContainer" containerID="b4c5a526f4da2db04711b6287428a2d9812139314c8f8f20342887f00fc1bba9" Nov 03 02:23:43.971310 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:23:43.971290 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4c5a526f4da2db04711b6287428a2d9812139314c8f8f20342887f00fc1bba9\": container with ID starting with b4c5a526f4da2db04711b6287428a2d9812139314c8f8f20342887f00fc1bba9 not found: ID does not exist" containerID="b4c5a526f4da2db04711b6287428a2d9812139314c8f8f20342887f00fc1bba9" Nov 03 02:23:43.971356 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:43.971318 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4c5a526f4da2db04711b6287428a2d9812139314c8f8f20342887f00fc1bba9"} err="failed to get container status \"b4c5a526f4da2db04711b6287428a2d9812139314c8f8f20342887f00fc1bba9\": rpc error: code = NotFound desc = could not find container \"b4c5a526f4da2db04711b6287428a2d9812139314c8f8f20342887f00fc1bba9\": container with ID starting with b4c5a526f4da2db04711b6287428a2d9812139314c8f8f20342887f00fc1bba9 not found: ID does not exist" Nov 03 02:23:44.026544 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:44.026448 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:23:44.035208 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:23:44.035175 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 02:23:44.131270 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:44.131234 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-3fb07e-7678cbbc5-kpm8s"] Nov 03 02:23:44.135351 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:44.135320 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-3fb07e-7678cbbc5-kpm8s"] Nov 03 02:23:44.229598 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:44.229549 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:23:45.072259 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:45.072224 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:23:46.089080 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:46.089048 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db26a4c8-466a-4c1e-ba51-30bd6a076791" path="/var/lib/kubelet/pods/db26a4c8-466a-4c1e-ba51-30bd6a076791/volumes" Nov 03 02:23:47.081403 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:47.079932 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:23:47.340557 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:47.340403 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:23:48.130898 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:48.130861 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-7ff5cb8dc-82dcd" podUID="11c05ff6-f766-450d-87f0-a83013d3183c" containerName="payload-processing" containerID="cri-o://8148ab73c172214f2794fbd2adfeb5d119f563580918059329b917dbb9e51391" gracePeriod=25 Nov 03 02:23:48.415177 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:48.415151 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7ff5cb8dc-82dcd" Nov 03 02:23:48.517212 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:48.517180 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11c05ff6-f766-450d-87f0-a83013d3183c-provider-credentials\" (UniqueName: \"kubernetes.io/projected/11c05ff6-f766-450d-87f0-a83013d3183c-provider-credentials\") pod \"11c05ff6-f766-450d-87f0-a83013d3183c\" (UID: \"11c05ff6-f766-450d-87f0-a83013d3183c\") " Nov 03 02:23:48.517430 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:48.517218 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11c05ff6-f766-450d-87f0-a83013d3183c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11c05ff6-f766-450d-87f0-a83013d3183c-plugins-config-volume\") pod \"11c05ff6-f766-450d-87f0-a83013d3183c\" (UID: \"11c05ff6-f766-450d-87f0-a83013d3183c\") " Nov 03 02:23:48.517430 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:48.517239 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11c05ff6-f766-450d-87f0-a83013d3183c-kube-api-access-wvjhh\" (UniqueName: \"kubernetes.io/projected/11c05ff6-f766-450d-87f0-a83013d3183c-kube-api-access-wvjhh\") pod \"11c05ff6-f766-450d-87f0-a83013d3183c\" (UID: \"11c05ff6-f766-450d-87f0-a83013d3183c\") " Nov 03 02:23:48.517430 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:48.517324 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11c05ff6-f766-450d-87f0-a83013d3183c-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/11c05ff6-f766-450d-87f0-a83013d3183c-routing-overlay-volume\") pod \"11c05ff6-f766-450d-87f0-a83013d3183c\" (UID: \"11c05ff6-f766-450d-87f0-a83013d3183c\") " Nov 03 02:23:48.517736 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:48.517705 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11c05ff6-f766-450d-87f0-a83013d3183c-routing-overlay-volume" pod "11c05ff6-f766-450d-87f0-a83013d3183c" (UID: "11c05ff6-f766-450d-87f0-a83013d3183c"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:23:48.517736 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:48.517713 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11c05ff6-f766-450d-87f0-a83013d3183c-plugins-config-volume" pod "11c05ff6-f766-450d-87f0-a83013d3183c" (UID: "11c05ff6-f766-450d-87f0-a83013d3183c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:23:48.517926 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:48.517910 5437 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11c05ff6-f766-450d-87f0-a83013d3183c-plugins-config-volume\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:23:48.517978 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:48.517931 5437 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/11c05ff6-f766-450d-87f0-a83013d3183c-routing-overlay-volume\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:23:48.519611 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:48.519587 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11c05ff6-f766-450d-87f0-a83013d3183c-provider-credentials" pod "11c05ff6-f766-450d-87f0-a83013d3183c" (UID: "11c05ff6-f766-450d-87f0-a83013d3183c"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:23:48.519705 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:48.519610 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11c05ff6-f766-450d-87f0-a83013d3183c-kube-api-access-wvjhh" pod "11c05ff6-f766-450d-87f0-a83013d3183c" (UID: "11c05ff6-f766-450d-87f0-a83013d3183c"). InnerVolumeSpecName "kube-api-access-wvjhh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:23:48.619634 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:48.619602 5437 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/11c05ff6-f766-450d-87f0-a83013d3183c-provider-credentials\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:23:48.619634 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:48.619627 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvjhh\" (UniqueName: \"kubernetes.io/projected/11c05ff6-f766-450d-87f0-a83013d3183c-kube-api-access-wvjhh\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:23:48.648475 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:48.648359 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:23:48.849714 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:48.849664 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:23:49.002941 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:49.002842 5437 generic.go:356] "Generic (PLEG): container finished" podID="11c05ff6-f766-450d-87f0-a83013d3183c" containerID="8148ab73c172214f2794fbd2adfeb5d119f563580918059329b917dbb9e51391" exitCode=0 Nov 03 02:23:49.002941 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:49.002911 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7ff5cb8dc-82dcd" event={"ID":"11c05ff6-f766-450d-87f0-a83013d3183c","Type":"ContainerDied","Data":"8148ab73c172214f2794fbd2adfeb5d119f563580918059329b917dbb9e51391"} Nov 03 02:23:49.003160 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:49.002946 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7ff5cb8dc-82dcd" Nov 03 02:23:49.003160 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:49.002985 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7ff5cb8dc-82dcd" event={"ID":"11c05ff6-f766-450d-87f0-a83013d3183c","Type":"ContainerDied","Data":"b7739f7e01617192951f49630eaa9fb06d1768fac36032ca553c4476d6b31683"} Nov 03 02:23:49.003160 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:49.003011 5437 scope.go:122] "RemoveContainer" containerID="8148ab73c172214f2794fbd2adfeb5d119f563580918059329b917dbb9e51391" Nov 03 02:23:49.015251 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:49.015228 5437 scope.go:122] "RemoveContainer" containerID="8148ab73c172214f2794fbd2adfeb5d119f563580918059329b917dbb9e51391" Nov 03 02:23:49.015573 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:23:49.015555 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8148ab73c172214f2794fbd2adfeb5d119f563580918059329b917dbb9e51391\": container with ID starting with 8148ab73c172214f2794fbd2adfeb5d119f563580918059329b917dbb9e51391 not found: ID does not exist" containerID="8148ab73c172214f2794fbd2adfeb5d119f563580918059329b917dbb9e51391" Nov 03 02:23:49.015630 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:49.015581 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8148ab73c172214f2794fbd2adfeb5d119f563580918059329b917dbb9e51391"} err="failed to get container status \"8148ab73c172214f2794fbd2adfeb5d119f563580918059329b917dbb9e51391\": rpc error: code = NotFound desc = could not find container \"8148ab73c172214f2794fbd2adfeb5d119f563580918059329b917dbb9e51391\": container with ID starting with 8148ab73c172214f2794fbd2adfeb5d119f563580918059329b917dbb9e51391 not found: ID does not exist" Nov 03 02:23:49.174276 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:49.174244 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7ff5cb8dc-82dcd"] Nov 03 02:23:49.182471 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:49.182435 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-7ff5cb8dc-82dcd"] Nov 03 02:23:49.275123 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:49.275039 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:23:50.089525 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:50.089494 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11c05ff6-f766-450d-87f0-a83013d3183c" path="/var/lib/kubelet/pods/11c05ff6-f766-450d-87f0-a83013d3183c/volumes" Nov 03 02:23:50.224471 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:50.224407 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:23:50.424024 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:50.423945 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:23:54.046299 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:54.046267 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:23:54.246024 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:54.245985 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:23:54.397765 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:54.397731 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d6fbd7-849f757bff-r85rr"] Nov 03 02:23:54.398326 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:54.398313 5437 cpu_manager.go:414] "containerMap: removing container" podUID="a5837de8-ead2-4681-8689-3d5fb3bd3ae1" containerName="istio-proxy" Nov 03 02:23:54.398405 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:54.398328 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5837de8-ead2-4681-8689-3d5fb3bd3ae1" containerName="istio-proxy" Nov 03 02:23:54.398405 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:54.398346 5437 cpu_manager.go:414] "containerMap: removing container" podUID="11c05ff6-f766-450d-87f0-a83013d3183c" containerName="payload-processing" Nov 03 02:23:54.398405 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:54.398351 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11c05ff6-f766-450d-87f0-a83013d3183c" containerName="payload-processing" Nov 03 02:23:54.398405 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:54.398390 5437 cpu_manager.go:414] "containerMap: removing container" podUID="db26a4c8-466a-4c1e-ba51-30bd6a076791" containerName="payload-processing" Nov 03 02:23:54.398405 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:54.398399 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db26a4c8-466a-4c1e-ba51-30bd6a076791" containerName="payload-processing" Nov 03 02:23:54.398568 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:54.398519 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5837de8-ead2-4681-8689-3d5fb3bd3ae1" containerName="istio-proxy" Nov 03 02:23:54.398568 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:54.398528 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="11c05ff6-f766-450d-87f0-a83013d3183c" containerName="payload-processing" Nov 03 02:23:54.398568 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:54.398535 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="db26a4c8-466a-4c1e-ba51-30bd6a076791" containerName="payload-processing" Nov 03 02:23:54.403219 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:54.403191 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d6fbd7-849f757bff-r85rr" Nov 03 02:23:54.466080 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:54.465963 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d6fbd7-849f757bff-r85rr"] Nov 03 02:23:54.499591 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:54.498830 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96tvg\" (UniqueName: \"kubernetes.io/projected/47743531-fbba-44ab-82a9-42b69510b4fe-kube-api-access-96tvg\") pod \"payload-pre-processing-e2e-inf-a-d6fbd7-849f757bff-r85rr\" (UID: \"47743531-fbba-44ab-82a9-42b69510b4fe\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d6fbd7-849f757bff-r85rr" Nov 03 02:23:54.499591 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:54.498900 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47743531-fbba-44ab-82a9-42b69510b4fe-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-d6fbd7-849f757bff-r85rr\" (UID: \"47743531-fbba-44ab-82a9-42b69510b4fe\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d6fbd7-849f757bff-r85rr" Nov 03 02:23:54.506796 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:54.506763 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-d6fbd7-dockercfg-6x4cw\"" Nov 03 02:23:54.506960 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:54.506877 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-d6fbd7\"" Nov 03 02:23:54.600704 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:54.600664 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96tvg\" (UniqueName: \"kubernetes.io/projected/47743531-fbba-44ab-82a9-42b69510b4fe-kube-api-access-96tvg\") pod \"payload-pre-processing-e2e-inf-a-d6fbd7-849f757bff-r85rr\" (UID: \"47743531-fbba-44ab-82a9-42b69510b4fe\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d6fbd7-849f757bff-r85rr" Nov 03 02:23:54.600704 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:54.600708 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47743531-fbba-44ab-82a9-42b69510b4fe-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-d6fbd7-849f757bff-r85rr\" (UID: \"47743531-fbba-44ab-82a9-42b69510b4fe\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d6fbd7-849f757bff-r85rr" Nov 03 02:23:54.601478 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:54.601451 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47743531-fbba-44ab-82a9-42b69510b4fe-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-d6fbd7-849f757bff-r85rr\" (UID: \"47743531-fbba-44ab-82a9-42b69510b4fe\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d6fbd7-849f757bff-r85rr" Nov 03 02:23:54.663813 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:54.663741 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96tvg\" (UniqueName: \"kubernetes.io/projected/47743531-fbba-44ab-82a9-42b69510b4fe-kube-api-access-96tvg\") pod \"payload-pre-processing-e2e-inf-a-d6fbd7-849f757bff-r85rr\" (UID: \"47743531-fbba-44ab-82a9-42b69510b4fe\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d6fbd7-849f757bff-r85rr" Nov 03 02:23:54.717549 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:54.717518 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d6fbd7-849f757bff-r85rr" Nov 03 02:23:55.110718 ip-10-0-136-16 kubenswrapper[5437]: W1103 02:23:55.110688 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47743531_fbba_44ab_82a9_42b69510b4fe.slice/crio-d90b7c7d42828279a366b80b2a3ef1e5c6b195f220f6e5459a748d50595a06bb WatchSource:0}: Error finding container d90b7c7d42828279a366b80b2a3ef1e5c6b195f220f6e5459a748d50595a06bb: Status 404 returned error can't find the container with id d90b7c7d42828279a366b80b2a3ef1e5c6b195f220f6e5459a748d50595a06bb Nov 03 02:23:55.113807 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:55.113785 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d6fbd7-849f757bff-r85rr"] Nov 03 02:23:55.397694 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:55.397594 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:23:55.599450 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:55.599409 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:23:56.058710 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:56.058666 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d6fbd7-849f757bff-r85rr" event={"ID":"47743531-fbba-44ab-82a9-42b69510b4fe","Type":"ContainerStarted","Data":"08582c8a4986e83c7e6c6fd74c4079cf55afb72cf01273106ca016403471dcc3"} Nov 03 02:23:56.058710 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:56.058739 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d6fbd7-849f757bff-r85rr" event={"ID":"47743531-fbba-44ab-82a9-42b69510b4fe","Type":"ContainerStarted","Data":"d90b7c7d42828279a366b80b2a3ef1e5c6b195f220f6e5459a748d50595a06bb"} Nov 03 02:23:56.058998 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:56.058979 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d6fbd7-849f757bff-r85rr" Nov 03 02:23:56.181276 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:56.181221 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d6fbd7-849f757bff-r85rr" podStartSLOduration=2.181205163 podStartE2EDuration="2.181205163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:23:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:23:56.178223993 +0000 UTC m=+3918.207706290" watchObservedRunningTime="2026-11-03 02:23:56.181205163 +0000 UTC m=+3918.210687457" Nov 03 02:23:56.508337 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:56.508247 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:23:56.711828 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:56.711792 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:23:58.069317 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:58.069260 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:23:59.878196 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:59.878158 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-ea5c75-5858f8984d-7xnbm"] Nov 03 02:23:59.882636 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:59.882608 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ea5c75-5858f8984d-7xnbm" Nov 03 02:23:59.936822 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:59.936790 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-ea5c75-5858f8984d-7xnbm"] Nov 03 02:23:59.969270 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:59.969226 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a22e3efc-1ac5-454a-9af2-7526a058bac6-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-ea5c75-5858f8984d-7xnbm\" (UID: \"a22e3efc-1ac5-454a-9af2-7526a058bac6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ea5c75-5858f8984d-7xnbm" Nov 03 02:23:59.969440 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:59.969356 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwqmd\" (UniqueName: \"kubernetes.io/projected/a22e3efc-1ac5-454a-9af2-7526a058bac6-kube-api-access-nwqmd\") pod \"maas-api-e2e-extmod-a-ea5c75-5858f8984d-7xnbm\" (UID: \"a22e3efc-1ac5-454a-9af2-7526a058bac6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ea5c75-5858f8984d-7xnbm" Nov 03 02:23:59.969440 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:59.969431 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a22e3efc-1ac5-454a-9af2-7526a058bac6-maas-api-tls\") pod \"maas-api-e2e-extmod-a-ea5c75-5858f8984d-7xnbm\" (UID: \"a22e3efc-1ac5-454a-9af2-7526a058bac6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ea5c75-5858f8984d-7xnbm" Nov 03 02:23:59.969571 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:59.969559 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a22e3efc-1ac5-454a-9af2-7526a058bac6-metrics-certs\") pod \"maas-api-e2e-extmod-a-ea5c75-5858f8984d-7xnbm\" (UID: \"a22e3efc-1ac5-454a-9af2-7526a058bac6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ea5c75-5858f8984d-7xnbm" Nov 03 02:23:59.985827 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:23:59.985793 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-ea5c75-serving-cert\"" Nov 03 02:24:00.070638 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:00.070599 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a22e3efc-1ac5-454a-9af2-7526a058bac6-metrics-certs\") pod \"maas-api-e2e-extmod-a-ea5c75-5858f8984d-7xnbm\" (UID: \"a22e3efc-1ac5-454a-9af2-7526a058bac6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ea5c75-5858f8984d-7xnbm" Nov 03 02:24:00.070817 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:00.070680 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a22e3efc-1ac5-454a-9af2-7526a058bac6-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-ea5c75-5858f8984d-7xnbm\" (UID: \"a22e3efc-1ac5-454a-9af2-7526a058bac6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ea5c75-5858f8984d-7xnbm" Nov 03 02:24:00.070817 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:00.070763 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwqmd\" (UniqueName: \"kubernetes.io/projected/a22e3efc-1ac5-454a-9af2-7526a058bac6-kube-api-access-nwqmd\") pod \"maas-api-e2e-extmod-a-ea5c75-5858f8984d-7xnbm\" (UID: \"a22e3efc-1ac5-454a-9af2-7526a058bac6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ea5c75-5858f8984d-7xnbm" Nov 03 02:24:00.070817 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:00.070811 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a22e3efc-1ac5-454a-9af2-7526a058bac6-maas-api-tls\") pod \"maas-api-e2e-extmod-a-ea5c75-5858f8984d-7xnbm\" (UID: \"a22e3efc-1ac5-454a-9af2-7526a058bac6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ea5c75-5858f8984d-7xnbm" Nov 03 02:24:00.070999 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:24:00.070946 5437 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-ea5c75-serving-cert: secret "maas-api-e2e-extmod-a-ea5c75-serving-cert" not found Nov 03 02:24:00.071056 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:24:00.071027 5437 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a22e3efc-1ac5-454a-9af2-7526a058bac6-maas-api-tls podName:a22e3efc-1ac5-454a-9af2-7526a058bac6 nodeName:}" failed. No retries permitted until 2026-11-03 02:24:00.571006287 +0000 UTC m=+3922.600488576 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a22e3efc-1ac5-454a-9af2-7526a058bac6-maas-api-tls") pod "maas-api-e2e-extmod-a-ea5c75-5858f8984d-7xnbm" (UID: "a22e3efc-1ac5-454a-9af2-7526a058bac6") : secret "maas-api-e2e-extmod-a-ea5c75-serving-cert" not found Nov 03 02:24:00.071430 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:00.071407 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a22e3efc-1ac5-454a-9af2-7526a058bac6-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-ea5c75-5858f8984d-7xnbm\" (UID: \"a22e3efc-1ac5-454a-9af2-7526a058bac6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ea5c75-5858f8984d-7xnbm" Nov 03 02:24:00.073160 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:00.073129 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a22e3efc-1ac5-454a-9af2-7526a058bac6-metrics-certs\") pod \"maas-api-e2e-extmod-a-ea5c75-5858f8984d-7xnbm\" (UID: \"a22e3efc-1ac5-454a-9af2-7526a058bac6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ea5c75-5858f8984d-7xnbm" Nov 03 02:24:00.126287 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:00.126244 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:24:00.135916 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:00.135849 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwqmd\" (UniqueName: \"kubernetes.io/projected/a22e3efc-1ac5-454a-9af2-7526a058bac6-kube-api-access-nwqmd\") pod \"maas-api-e2e-extmod-a-ea5c75-5858f8984d-7xnbm\" (UID: \"a22e3efc-1ac5-454a-9af2-7526a058bac6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ea5c75-5858f8984d-7xnbm" Nov 03 02:24:00.332519 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:00.332471 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:24:00.530718 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:00.530638 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:24:00.578453 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:00.578410 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a22e3efc-1ac5-454a-9af2-7526a058bac6-maas-api-tls\") pod \"maas-api-e2e-extmod-a-ea5c75-5858f8984d-7xnbm\" (UID: \"a22e3efc-1ac5-454a-9af2-7526a058bac6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ea5c75-5858f8984d-7xnbm" Nov 03 02:24:00.580882 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:00.580853 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a22e3efc-1ac5-454a-9af2-7526a058bac6-maas-api-tls\") pod \"maas-api-e2e-extmod-a-ea5c75-5858f8984d-7xnbm\" (UID: \"a22e3efc-1ac5-454a-9af2-7526a058bac6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ea5c75-5858f8984d-7xnbm" Nov 03 02:24:00.800980 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:00.800948 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ea5c75-5858f8984d-7xnbm" Nov 03 02:24:01.172034 ip-10-0-136-16 kubenswrapper[5437]: W1103 02:24:01.171996 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda22e3efc_1ac5_454a_9af2_7526a058bac6.slice/crio-e9eae570f969fe1dfc25bb9013a62e8443b8d32e56802c1eb9de1896d74b7f30 WatchSource:0}: Error finding container e9eae570f969fe1dfc25bb9013a62e8443b8d32e56802c1eb9de1896d74b7f30: Status 404 returned error can't find the container with id e9eae570f969fe1dfc25bb9013a62e8443b8d32e56802c1eb9de1896d74b7f30 Nov 03 02:24:01.173519 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:01.173496 5437 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 02:24:01.175374 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:01.175339 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-ea5c75-5858f8984d-7xnbm"] Nov 03 02:24:02.097054 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:02.097019 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ea5c75-5858f8984d-7xnbm" event={"ID":"a22e3efc-1ac5-454a-9af2-7526a058bac6","Type":"ContainerStarted","Data":"e9eae570f969fe1dfc25bb9013a62e8443b8d32e56802c1eb9de1896d74b7f30"} Nov 03 02:24:02.960715 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:02.960678 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:24:03.117957 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:03.117918 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ea5c75-5858f8984d-7xnbm" event={"ID":"a22e3efc-1ac5-454a-9af2-7526a058bac6","Type":"ContainerStarted","Data":"d6c09a018a8ee13e3e5d79bba06cb2107251ad27618eb493cd41f5afd2bc1624"} Nov 03 02:24:03.118192 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:03.118009 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ea5c75-5858f8984d-7xnbm" Nov 03 02:24:03.218605 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:03.218522 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:24:03.241679 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:03.241626 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ea5c75-5858f8984d-7xnbm" podStartSLOduration=3.211352684 podStartE2EDuration="4.241611941s" totalImagesPullingTime="1.030259257s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:23:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:24:03.239169116 +0000 UTC m=+3925.268651425" watchObservedRunningTime="2026-11-03 02:24:03.241611941 +0000 UTC m=+3925.271094235" Nov 03 02:24:03.423537 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:03.423479 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:24:03.648636 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:03.648595 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:24:05.067741 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:05.067699 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:24:05.302199 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:05.302167 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:24:06.459620 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:06.459583 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:24:06.663039 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:06.662980 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:24:07.068718 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:07.068679 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d6fbd7-849f757bff-r85rr" Nov 03 02:24:07.167910 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:07.167871 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:24:09.131675 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:09.131646 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ea5c75-5858f8984d-7xnbm" Nov 03 02:24:09.876350 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:09.876312 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:24:10.848662 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:10.848611 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:24:11.052858 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:11.052818 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:24:13.966089 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:13.966045 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:24:14.049621 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:24:14.049579 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 02:24:14.168448 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:14.168414 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:24:15.222013 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:15.221973 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:24:20.179082 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:20.179041 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:24:20.190563 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:20.190527 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-rrtbq"] Nov 03 02:24:20.194811 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:20.194793 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-rrtbq" Nov 03 02:24:20.220545 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:20.220514 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/250a28b0-9884-43a0-9991-61fb8e3a5be4-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-rrtbq\" (UID: \"250a28b0-9884-43a0-9991-61fb8e3a5be4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-rrtbq" Nov 03 02:24:20.220689 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:20.220550 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/250a28b0-9884-43a0-9991-61fb8e3a5be4-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-rrtbq\" (UID: \"250a28b0-9884-43a0-9991-61fb8e3a5be4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-rrtbq" Nov 03 02:24:20.220741 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:20.220703 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/250a28b0-9884-43a0-9991-61fb8e3a5be4-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-rrtbq\" (UID: \"250a28b0-9884-43a0-9991-61fb8e3a5be4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-rrtbq" Nov 03 02:24:20.220791 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:20.220762 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9fq7\" (UniqueName: \"kubernetes.io/projected/250a28b0-9884-43a0-9991-61fb8e3a5be4-kube-api-access-r9fq7\") pod \"maas-default-gateway-openshift-default-59f957fdd7-rrtbq\" (UID: \"250a28b0-9884-43a0-9991-61fb8e3a5be4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-rrtbq" Nov 03 02:24:20.220885 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:20.220815 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/250a28b0-9884-43a0-9991-61fb8e3a5be4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-rrtbq\" (UID: \"250a28b0-9884-43a0-9991-61fb8e3a5be4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-rrtbq" Nov 03 02:24:20.220885 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:20.220869 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/250a28b0-9884-43a0-9991-61fb8e3a5be4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-rrtbq\" (UID: \"250a28b0-9884-43a0-9991-61fb8e3a5be4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-rrtbq" Nov 03 02:24:20.220989 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:20.220890 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/250a28b0-9884-43a0-9991-61fb8e3a5be4-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-rrtbq\" (UID: \"250a28b0-9884-43a0-9991-61fb8e3a5be4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-rrtbq" Nov 03 02:24:20.220989 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:20.220976 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/250a28b0-9884-43a0-9991-61fb8e3a5be4-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-rrtbq\" (UID: \"250a28b0-9884-43a0-9991-61fb8e3a5be4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-rrtbq" Nov 03 02:24:20.221101 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:20.221011 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/250a28b0-9884-43a0-9991-61fb8e3a5be4-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-rrtbq\" (UID: \"250a28b0-9884-43a0-9991-61fb8e3a5be4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-rrtbq" Nov 03 02:24:20.284707 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:20.284665 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-rrtbq"] Nov 03 02:24:20.321842 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:20.321807 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/250a28b0-9884-43a0-9991-61fb8e3a5be4-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-rrtbq\" (UID: \"250a28b0-9884-43a0-9991-61fb8e3a5be4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-rrtbq" Nov 03 02:24:20.322038 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:20.321857 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/250a28b0-9884-43a0-9991-61fb8e3a5be4-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-rrtbq\" (UID: \"250a28b0-9884-43a0-9991-61fb8e3a5be4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-rrtbq" Nov 03 02:24:20.322038 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:20.321879 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/250a28b0-9884-43a0-9991-61fb8e3a5be4-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-rrtbq\" (UID: \"250a28b0-9884-43a0-9991-61fb8e3a5be4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-rrtbq" Nov 03 02:24:20.322038 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:20.321941 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/250a28b0-9884-43a0-9991-61fb8e3a5be4-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-rrtbq\" (UID: \"250a28b0-9884-43a0-9991-61fb8e3a5be4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-rrtbq" Nov 03 02:24:20.322038 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:20.321972 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/250a28b0-9884-43a0-9991-61fb8e3a5be4-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-rrtbq\" (UID: \"250a28b0-9884-43a0-9991-61fb8e3a5be4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-rrtbq" Nov 03 02:24:20.322293 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:20.322074 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/250a28b0-9884-43a0-9991-61fb8e3a5be4-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-rrtbq\" (UID: \"250a28b0-9884-43a0-9991-61fb8e3a5be4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-rrtbq" Nov 03 02:24:20.322293 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:20.322145 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9fq7\" (UniqueName: \"kubernetes.io/projected/250a28b0-9884-43a0-9991-61fb8e3a5be4-kube-api-access-r9fq7\") pod \"maas-default-gateway-openshift-default-59f957fdd7-rrtbq\" (UID: \"250a28b0-9884-43a0-9991-61fb8e3a5be4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-rrtbq" Nov 03 02:24:20.322293 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:20.322235 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/250a28b0-9884-43a0-9991-61fb8e3a5be4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-rrtbq\" (UID: \"250a28b0-9884-43a0-9991-61fb8e3a5be4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-rrtbq" Nov 03 02:24:20.322480 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:20.322303 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/250a28b0-9884-43a0-9991-61fb8e3a5be4-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-rrtbq\" (UID: \"250a28b0-9884-43a0-9991-61fb8e3a5be4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-rrtbq" Nov 03 02:24:20.322480 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:20.322329 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/250a28b0-9884-43a0-9991-61fb8e3a5be4-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-rrtbq\" (UID: \"250a28b0-9884-43a0-9991-61fb8e3a5be4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-rrtbq" Nov 03 02:24:20.322480 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:20.322356 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/250a28b0-9884-43a0-9991-61fb8e3a5be4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-rrtbq\" (UID: \"250a28b0-9884-43a0-9991-61fb8e3a5be4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-rrtbq" Nov 03 02:24:20.322480 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:20.322458 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/250a28b0-9884-43a0-9991-61fb8e3a5be4-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-rrtbq\" (UID: \"250a28b0-9884-43a0-9991-61fb8e3a5be4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-rrtbq" Nov 03 02:24:20.322618 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:20.322492 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/250a28b0-9884-43a0-9991-61fb8e3a5be4-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-rrtbq\" (UID: \"250a28b0-9884-43a0-9991-61fb8e3a5be4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-rrtbq" Nov 03 02:24:20.322890 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:20.322861 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/250a28b0-9884-43a0-9991-61fb8e3a5be4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-rrtbq\" (UID: \"250a28b0-9884-43a0-9991-61fb8e3a5be4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-rrtbq" Nov 03 02:24:20.324557 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:20.324530 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/250a28b0-9884-43a0-9991-61fb8e3a5be4-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-rrtbq\" (UID: \"250a28b0-9884-43a0-9991-61fb8e3a5be4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-rrtbq" Nov 03 02:24:20.324672 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:20.324651 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/250a28b0-9884-43a0-9991-61fb8e3a5be4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-rrtbq\" (UID: \"250a28b0-9884-43a0-9991-61fb8e3a5be4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-rrtbq" Nov 03 02:24:20.384949 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:20.384911 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/250a28b0-9884-43a0-9991-61fb8e3a5be4-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-rrtbq\" (UID: \"250a28b0-9884-43a0-9991-61fb8e3a5be4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-rrtbq" Nov 03 02:24:20.385134 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:20.385024 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9fq7\" (UniqueName: \"kubernetes.io/projected/250a28b0-9884-43a0-9991-61fb8e3a5be4-kube-api-access-r9fq7\") pod \"maas-default-gateway-openshift-default-59f957fdd7-rrtbq\" (UID: \"250a28b0-9884-43a0-9991-61fb8e3a5be4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-rrtbq" Nov 03 02:24:20.508671 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:20.508558 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-rrtbq" Nov 03 02:24:20.929802 ip-10-0-136-16 kubenswrapper[5437]: W1103 02:24:20.929737 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod250a28b0_9884_43a0_9991_61fb8e3a5be4.slice/crio-d47089d78a80c4a091785d560e6d0cbe68b71c86139f3e4bb1f2e41087936e1a WatchSource:0}: Error finding container d47089d78a80c4a091785d560e6d0cbe68b71c86139f3e4bb1f2e41087936e1a: Status 404 returned error can't find the container with id d47089d78a80c4a091785d560e6d0cbe68b71c86139f3e4bb1f2e41087936e1a Nov 03 02:24:20.932417 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:20.932360 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 02:24:20.932510 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:20.932454 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 02:24:20.932510 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:20.932497 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 02:24:20.933211 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:20.933165 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-rrtbq"] Nov 03 02:24:21.246115 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:21.245992 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-rrtbq" event={"ID":"250a28b0-9884-43a0-9991-61fb8e3a5be4","Type":"ContainerStarted","Data":"e0b7a6b45540fef8d71259fad1763d12270698758371173497f6f422097c8334"} Nov 03 02:24:21.246115 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:21.246054 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-rrtbq" event={"ID":"250a28b0-9884-43a0-9991-61fb8e3a5be4","Type":"ContainerStarted","Data":"d47089d78a80c4a091785d560e6d0cbe68b71c86139f3e4bb1f2e41087936e1a"} Nov 03 02:24:21.372874 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:21.372810 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-rrtbq" podStartSLOduration=1.3727897900000001 podStartE2EDuration="1.37278979s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:24:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:24:21.370848686 +0000 UTC m=+3943.400330981" watchObservedRunningTime="2026-11-03 02:24:21.37278979 +0000 UTC m=+3943.402272086" Nov 03 02:24:21.513443 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:21.513259 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-rrtbq" Nov 03 02:24:21.515216 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:21.515179 5437 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-rrtbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body= Nov 03 02:24:21.515561 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:21.515241 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-rrtbq" podUID="250a28b0-9884-43a0-9991-61fb8e3a5be4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" Nov 03 02:24:22.509458 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:22.509419 5437 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-rrtbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body= Nov 03 02:24:22.509873 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:22.509485 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-rrtbq" podUID="250a28b0-9884-43a0-9991-61fb8e3a5be4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" Nov 03 02:24:23.509842 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:23.509803 5437 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-rrtbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body= Nov 03 02:24:23.510359 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:23.509869 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-rrtbq" podUID="250a28b0-9884-43a0-9991-61fb8e3a5be4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" Nov 03 02:24:24.508888 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:24.508860 5437 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-rrtbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body= Nov 03 02:24:24.509142 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:24.509121 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-rrtbq" podUID="250a28b0-9884-43a0-9991-61fb8e3a5be4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" Nov 03 02:24:25.259066 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:25.259025 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:24:25.509322 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:25.509243 5437 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-rrtbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body= Nov 03 02:24:25.509517 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:25.509440 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-rrtbq" podUID="250a28b0-9884-43a0-9991-61fb8e3a5be4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" Nov 03 02:24:26.509089 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:26.509055 5437 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-rrtbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body= Nov 03 02:24:26.509504 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:26.509104 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-rrtbq" podUID="250a28b0-9884-43a0-9991-61fb8e3a5be4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" Nov 03 02:24:27.509878 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:27.509787 5437 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-rrtbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body= Nov 03 02:24:27.510255 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:27.509867 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-rrtbq" podUID="250a28b0-9884-43a0-9991-61fb8e3a5be4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" Nov 03 02:24:28.509641 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:28.509600 5437 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-rrtbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body= Nov 03 02:24:28.509825 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:28.509687 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-rrtbq" podUID="250a28b0-9884-43a0-9991-61fb8e3a5be4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" Nov 03 02:24:29.671751 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:29.671692 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-rrtbq" Nov 03 02:24:29.672658 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:29.672635 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-rrtbq" Nov 03 02:24:29.673290 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:29.673271 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-rrtbq" Nov 03 02:24:30.262432 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:30.262391 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:24:31.424723 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:31.424694 5437 generic.go:356] "Generic (PLEG): container finished" podID="38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c" containerID="79d094231d716860f9c35455796545cebee39a73796351fa8f0816d924774e07" exitCode=0 Nov 03 02:24:31.425222 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:31.424768 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ea5c75-openshift-default-75dc4675bf-qbhjq" event={"ID":"38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c","Type":"ContainerDied","Data":"79d094231d716860f9c35455796545cebee39a73796351fa8f0816d924774e07"} Nov 03 02:24:31.425349 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:31.425317 5437 scope.go:122] "RemoveContainer" containerID="79d094231d716860f9c35455796545cebee39a73796351fa8f0816d924774e07" Nov 03 02:24:31.426937 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:31.426906 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 02:24:31.427052 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:31.426993 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 02:24:31.427052 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:31.427041 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 02:24:32.452670 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:32.452632 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ea5c75-openshift-default-75dc4675bf-qbhjq" event={"ID":"38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c","Type":"ContainerStarted","Data":"6803df855d7fe1bcfdeba838e0a7d23bc4ba21655ed60e3257b7ff94b84c06ba"} Nov 03 02:24:33.359145 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:33.359104 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-ea5c75-openshift-default-75dc4675bf-qbhjq" Nov 03 02:24:33.359962 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:33.359936 5437 patch_prober.go:29] interesting pod/e2e-extmod-a-ea5c75-openshift-default-75dc4675bf-qbhjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Nov 03 02:24:33.360043 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:33.359989 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ea5c75-openshift-default-75dc4675bf-qbhjq" podUID="38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Nov 03 02:24:34.358484 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:34.358452 5437 patch_prober.go:29] interesting pod/e2e-extmod-a-ea5c75-openshift-default-75dc4675bf-qbhjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Nov 03 02:24:34.358904 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:34.358503 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ea5c75-openshift-default-75dc4675bf-qbhjq" podUID="38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Nov 03 02:24:35.035389 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:35.031240 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:24:35.359080 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:35.359047 5437 patch_prober.go:29] interesting pod/e2e-extmod-a-ea5c75-openshift-default-75dc4675bf-qbhjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Nov 03 02:24:35.359492 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:35.359102 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ea5c75-openshift-default-75dc4675bf-qbhjq" podUID="38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Nov 03 02:24:36.358321 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:36.358281 5437 patch_prober.go:29] interesting pod/e2e-extmod-a-ea5c75-openshift-default-75dc4675bf-qbhjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Nov 03 02:24:36.358638 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:36.358342 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ea5c75-openshift-default-75dc4675bf-qbhjq" podUID="38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Nov 03 02:24:36.507749 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:36.507715 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-ea5c75-openshift-default-75dc4675bf-qbhjq" Nov 03 02:24:37.358488 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:37.358445 5437 patch_prober.go:29] interesting pod/e2e-extmod-a-ea5c75-openshift-default-75dc4675bf-qbhjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Nov 03 02:24:37.358700 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:37.358507 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ea5c75-openshift-default-75dc4675bf-qbhjq" podUID="38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Nov 03 02:24:38.363951 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:38.363919 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-ea5c75-openshift-default-75dc4675bf-qbhjq" Nov 03 02:24:38.364893 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:38.364870 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-ea5c75-openshift-default-75dc4675bf-qbhjq" Nov 03 02:24:39.273010 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:39.272976 5437 scope.go:122] "RemoveContainer" containerID="69f899906e8c170453bf9a7a42d26165c8df0a3ac53a5a98b28acb4d21b7cd47" Nov 03 02:24:39.926494 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:39.926452 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:24:44.036776 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:24:44.036738 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 02:24:44.624275 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:44.624232 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:24:49.393939 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:49.393844 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:24:50.423047 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:50.423014 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k42gq"] Nov 03 02:24:50.427246 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:50.427225 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k42gq" Nov 03 02:24:50.508787 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:50.508751 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/615071b1-f5f9-4d03-90b8-1210c450902d-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-k42gq\" (UID: \"615071b1-f5f9-4d03-90b8-1210c450902d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k42gq" Nov 03 02:24:50.508990 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:50.508841 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/615071b1-f5f9-4d03-90b8-1210c450902d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-k42gq\" (UID: \"615071b1-f5f9-4d03-90b8-1210c450902d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k42gq" Nov 03 02:24:50.508990 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:50.508939 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/615071b1-f5f9-4d03-90b8-1210c450902d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-k42gq\" (UID: \"615071b1-f5f9-4d03-90b8-1210c450902d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k42gq" Nov 03 02:24:50.509125 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:50.509044 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/615071b1-f5f9-4d03-90b8-1210c450902d-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-k42gq\" (UID: \"615071b1-f5f9-4d03-90b8-1210c450902d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k42gq" Nov 03 02:24:50.509209 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:50.509194 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/615071b1-f5f9-4d03-90b8-1210c450902d-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-k42gq\" (UID: \"615071b1-f5f9-4d03-90b8-1210c450902d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k42gq" Nov 03 02:24:50.509281 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:50.509265 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cp6n\" (UniqueName: \"kubernetes.io/projected/615071b1-f5f9-4d03-90b8-1210c450902d-kube-api-access-6cp6n\") pod \"maas-default-gateway-openshift-default-59f957fdd7-k42gq\" (UID: \"615071b1-f5f9-4d03-90b8-1210c450902d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k42gq" Nov 03 02:24:50.509343 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:50.509295 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/615071b1-f5f9-4d03-90b8-1210c450902d-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-k42gq\" (UID: \"615071b1-f5f9-4d03-90b8-1210c450902d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k42gq" Nov 03 02:24:50.509343 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:50.509324 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/615071b1-f5f9-4d03-90b8-1210c450902d-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-k42gq\" (UID: \"615071b1-f5f9-4d03-90b8-1210c450902d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k42gq" Nov 03 02:24:50.509467 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:50.509434 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/615071b1-f5f9-4d03-90b8-1210c450902d-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-k42gq\" (UID: \"615071b1-f5f9-4d03-90b8-1210c450902d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k42gq" Nov 03 02:24:50.521535 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:50.521505 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k42gq"] Nov 03 02:24:50.610798 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:50.610763 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cp6n\" (UniqueName: \"kubernetes.io/projected/615071b1-f5f9-4d03-90b8-1210c450902d-kube-api-access-6cp6n\") pod \"maas-default-gateway-openshift-default-59f957fdd7-k42gq\" (UID: \"615071b1-f5f9-4d03-90b8-1210c450902d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k42gq" Nov 03 02:24:50.610798 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:50.610800 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/615071b1-f5f9-4d03-90b8-1210c450902d-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-k42gq\" (UID: \"615071b1-f5f9-4d03-90b8-1210c450902d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k42gq" Nov 03 02:24:50.611026 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:50.610819 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/615071b1-f5f9-4d03-90b8-1210c450902d-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-k42gq\" (UID: \"615071b1-f5f9-4d03-90b8-1210c450902d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k42gq" Nov 03 02:24:50.611026 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:50.610889 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/615071b1-f5f9-4d03-90b8-1210c450902d-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-k42gq\" (UID: \"615071b1-f5f9-4d03-90b8-1210c450902d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k42gq" Nov 03 02:24:50.611026 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:50.610942 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/615071b1-f5f9-4d03-90b8-1210c450902d-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-k42gq\" (UID: \"615071b1-f5f9-4d03-90b8-1210c450902d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k42gq" Nov 03 02:24:50.611026 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:50.611011 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/615071b1-f5f9-4d03-90b8-1210c450902d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-k42gq\" (UID: \"615071b1-f5f9-4d03-90b8-1210c450902d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k42gq" Nov 03 02:24:50.611309 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:50.611068 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/615071b1-f5f9-4d03-90b8-1210c450902d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-k42gq\" (UID: \"615071b1-f5f9-4d03-90b8-1210c450902d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k42gq" Nov 03 02:24:50.611309 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:50.611146 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/615071b1-f5f9-4d03-90b8-1210c450902d-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-k42gq\" (UID: \"615071b1-f5f9-4d03-90b8-1210c450902d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k42gq" Nov 03 02:24:50.611309 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:50.611267 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/615071b1-f5f9-4d03-90b8-1210c450902d-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-k42gq\" (UID: \"615071b1-f5f9-4d03-90b8-1210c450902d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k42gq" Nov 03 02:24:50.611309 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:50.611272 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/615071b1-f5f9-4d03-90b8-1210c450902d-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-k42gq\" (UID: \"615071b1-f5f9-4d03-90b8-1210c450902d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k42gq" Nov 03 02:24:50.611574 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:50.611315 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/615071b1-f5f9-4d03-90b8-1210c450902d-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-k42gq\" (UID: \"615071b1-f5f9-4d03-90b8-1210c450902d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k42gq" Nov 03 02:24:50.611637 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:50.611616 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/615071b1-f5f9-4d03-90b8-1210c450902d-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-k42gq\" (UID: \"615071b1-f5f9-4d03-90b8-1210c450902d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k42gq" Nov 03 02:24:50.611758 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:50.611731 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/615071b1-f5f9-4d03-90b8-1210c450902d-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-k42gq\" (UID: \"615071b1-f5f9-4d03-90b8-1210c450902d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k42gq" Nov 03 02:24:50.611890 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:50.611791 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/615071b1-f5f9-4d03-90b8-1210c450902d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-k42gq\" (UID: \"615071b1-f5f9-4d03-90b8-1210c450902d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k42gq" Nov 03 02:24:50.613433 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:50.613408 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/615071b1-f5f9-4d03-90b8-1210c450902d-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-k42gq\" (UID: \"615071b1-f5f9-4d03-90b8-1210c450902d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k42gq" Nov 03 02:24:50.613728 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:50.613705 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/615071b1-f5f9-4d03-90b8-1210c450902d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-k42gq\" (UID: \"615071b1-f5f9-4d03-90b8-1210c450902d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k42gq" Nov 03 02:24:50.673859 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:50.673790 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cp6n\" (UniqueName: \"kubernetes.io/projected/615071b1-f5f9-4d03-90b8-1210c450902d-kube-api-access-6cp6n\") pod \"maas-default-gateway-openshift-default-59f957fdd7-k42gq\" (UID: \"615071b1-f5f9-4d03-90b8-1210c450902d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k42gq" Nov 03 02:24:50.673859 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:50.673799 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/615071b1-f5f9-4d03-90b8-1210c450902d-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-k42gq\" (UID: \"615071b1-f5f9-4d03-90b8-1210c450902d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k42gq" Nov 03 02:24:50.741941 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:50.741900 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k42gq" Nov 03 02:24:51.168032 ip-10-0-136-16 kubenswrapper[5437]: W1103 02:24:51.167998 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod615071b1_f5f9_4d03_90b8_1210c450902d.slice/crio-b3ba2279c7ef03f1832d6c523a37505ec0f41ecab5dfdd1cb4d6de22fabcbafb WatchSource:0}: Error finding container b3ba2279c7ef03f1832d6c523a37505ec0f41ecab5dfdd1cb4d6de22fabcbafb: Status 404 returned error can't find the container with id b3ba2279c7ef03f1832d6c523a37505ec0f41ecab5dfdd1cb4d6de22fabcbafb Nov 03 02:24:51.170803 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:51.170767 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 02:24:51.170912 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:51.170853 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 02:24:51.170912 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:51.170901 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 02:24:51.171271 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:51.171248 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k42gq"] Nov 03 02:24:51.573512 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:51.573474 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k42gq" event={"ID":"615071b1-f5f9-4d03-90b8-1210c450902d","Type":"ContainerStarted","Data":"6049cceacbaaff9610f640c7b31bc02e38872a068dce6da6c2782a5c48be6e06"} Nov 03 02:24:51.574010 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:51.573532 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k42gq" event={"ID":"615071b1-f5f9-4d03-90b8-1210c450902d","Type":"ContainerStarted","Data":"b3ba2279c7ef03f1832d6c523a37505ec0f41ecab5dfdd1cb4d6de22fabcbafb"} Nov 03 02:24:51.700456 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:51.700353 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k42gq" podStartSLOduration=1.7003350959999999 podStartE2EDuration="1.700335096s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:24:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:24:51.697231836 +0000 UTC m=+3973.726714131" watchObservedRunningTime="2026-11-03 02:24:51.700335096 +0000 UTC m=+3973.729817392" Nov 03 02:24:51.742912 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:51.742878 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k42gq" Nov 03 02:24:52.744672 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:52.744633 5437 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-k42gq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body= Nov 03 02:24:52.745132 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:52.744687 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k42gq" podUID="615071b1-f5f9-4d03-90b8-1210c450902d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" Nov 03 02:24:53.576135 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:53.576096 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:24:53.742779 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:53.742740 5437 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-k42gq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body= Nov 03 02:24:53.742991 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:53.742795 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k42gq" podUID="615071b1-f5f9-4d03-90b8-1210c450902d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" Nov 03 02:24:54.742805 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:54.742755 5437 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-k42gq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body= Nov 03 02:24:54.743333 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:54.742825 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k42gq" podUID="615071b1-f5f9-4d03-90b8-1210c450902d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" Nov 03 02:24:55.381869 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:55.381828 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:24:55.742907 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:55.742797 5437 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-k42gq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body= Nov 03 02:24:55.742907 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:55.742869 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k42gq" podUID="615071b1-f5f9-4d03-90b8-1210c450902d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" Nov 03 02:24:56.742772 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:56.742722 5437 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-k42gq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body= Nov 03 02:24:56.742772 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:56.742779 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k42gq" podUID="615071b1-f5f9-4d03-90b8-1210c450902d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" Nov 03 02:24:57.742309 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:57.742270 5437 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-k42gq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body= Nov 03 02:24:57.742511 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:57.742354 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k42gq" podUID="615071b1-f5f9-4d03-90b8-1210c450902d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" Nov 03 02:24:58.401107 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:58.401066 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:24:58.743295 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:58.743210 5437 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-k42gq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body= Nov 03 02:24:58.743295 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:58.743277 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k42gq" podUID="615071b1-f5f9-4d03-90b8-1210c450902d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" Nov 03 02:24:59.794345 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:59.794301 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k42gq" Nov 03 02:24:59.795674 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:59.795648 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k42gq" Nov 03 02:24:59.796482 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:24:59.796455 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k42gq" Nov 03 02:25:00.070001 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:25:00.069956 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:25:00.275095 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:25:00.275065 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:25:00.485514 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:25:00.485410 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:25:01.565042 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:25:01.564988 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:25:01.770613 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:25:01.770573 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:25:01.976658 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:25:01.976564 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:25:02.309146 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:25:02.309114 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:25:03.253178 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:25:03.253142 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:25:03.449439 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:25:03.449403 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:25:03.793200 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:25:03.793161 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:25:04.069686 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:25:04.069650 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:25:05.346126 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:25:05.346084 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:25:08.469518 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:25:08.469480 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:25:08.730475 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:25:08.730351 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:25:10.408358 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:25:10.408319 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:25:10.658534 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:25:10.658434 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:25:13.504226 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:25:13.504192 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:25:13.759975 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:25:13.759669 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:25:13.843571 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:25:13.843535 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0b98c1-7867fb5b9f-pnhpm"] Nov 03 02:25:13.847711 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:25:13.847687 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0b98c1-7867fb5b9f-pnhpm" Nov 03 02:25:13.902311 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:25:13.902276 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0b98c1-7867fb5b9f-pnhpm"] Nov 03 02:25:13.929825 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:25:13.929785 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/427bd62b-14cd-49e0-9b4e-4fa30b021ce1-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-0b98c1-7867fb5b9f-pnhpm\" (UID: \"427bd62b-14cd-49e0-9b4e-4fa30b021ce1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0b98c1-7867fb5b9f-pnhpm" Nov 03 02:25:13.930079 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:25:13.930058 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vn6cs\" (UniqueName: \"kubernetes.io/projected/427bd62b-14cd-49e0-9b4e-4fa30b021ce1-kube-api-access-vn6cs\") pod \"payload-pre-processing-e2e-inf-a-0b98c1-7867fb5b9f-pnhpm\" (UID: \"427bd62b-14cd-49e0-9b4e-4fa30b021ce1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0b98c1-7867fb5b9f-pnhpm" Nov 03 02:25:13.950078 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:25:13.950047 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-0b98c1\"" Nov 03 02:25:13.950078 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:25:13.950048 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-0b98c1-dockercfg-7dmx7\"" Nov 03 02:25:14.031828 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:25:14.031746 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vn6cs\" (UniqueName: \"kubernetes.io/projected/427bd62b-14cd-49e0-9b4e-4fa30b021ce1-kube-api-access-vn6cs\") pod \"payload-pre-processing-e2e-inf-a-0b98c1-7867fb5b9f-pnhpm\" (UID: \"427bd62b-14cd-49e0-9b4e-4fa30b021ce1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0b98c1-7867fb5b9f-pnhpm" Nov 03 02:25:14.031980 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:25:14.031849 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/427bd62b-14cd-49e0-9b4e-4fa30b021ce1-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-0b98c1-7867fb5b9f-pnhpm\" (UID: \"427bd62b-14cd-49e0-9b4e-4fa30b021ce1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0b98c1-7867fb5b9f-pnhpm" Nov 03 02:25:14.032807 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:25:14.032781 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/427bd62b-14cd-49e0-9b4e-4fa30b021ce1-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-0b98c1-7867fb5b9f-pnhpm\" (UID: \"427bd62b-14cd-49e0-9b4e-4fa30b021ce1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0b98c1-7867fb5b9f-pnhpm" Nov 03 02:25:14.046694 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:25:14.046654 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 02:25:14.095378 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:25:14.095337 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vn6cs\" (UniqueName: \"kubernetes.io/projected/427bd62b-14cd-49e0-9b4e-4fa30b021ce1-kube-api-access-vn6cs\") pod \"payload-pre-processing-e2e-inf-a-0b98c1-7867fb5b9f-pnhpm\" (UID: \"427bd62b-14cd-49e0-9b4e-4fa30b021ce1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0b98c1-7867fb5b9f-pnhpm" Nov 03 02:25:14.162261 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:25:14.162230 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0b98c1-7867fb5b9f-pnhpm" Nov 03 02:25:14.533075 ip-10-0-136-16 kubenswrapper[5437]: W1103 02:25:14.533046 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod427bd62b_14cd_49e0_9b4e_4fa30b021ce1.slice/crio-4e56ad5cc63c404443e646fae5056d4e4c2822e8e09381beee4138aeae713b4c WatchSource:0}: Error finding container 4e56ad5cc63c404443e646fae5056d4e4c2822e8e09381beee4138aeae713b4c: Status 404 returned error can't find the container with id 4e56ad5cc63c404443e646fae5056d4e4c2822e8e09381beee4138aeae713b4c Nov 03 02:25:14.538403 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:25:14.538236 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0b98c1-7867fb5b9f-pnhpm"] Nov 03 02:25:14.730137 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:25:14.730097 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0b98c1-7867fb5b9f-pnhpm" event={"ID":"427bd62b-14cd-49e0-9b4e-4fa30b021ce1","Type":"ContainerStarted","Data":"ed041cb7ca41f75af07e3032017115f2a66e7fa2f072f5899d9a4b1af9f8408f"} Nov 03 02:25:14.730304 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:25:14.730185 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0b98c1-7867fb5b9f-pnhpm" event={"ID":"427bd62b-14cd-49e0-9b4e-4fa30b021ce1","Type":"ContainerStarted","Data":"4e56ad5cc63c404443e646fae5056d4e4c2822e8e09381beee4138aeae713b4c"} Nov 03 02:25:14.730304 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:25:14.730214 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0b98c1-7867fb5b9f-pnhpm" Nov 03 02:25:14.850965 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:25:14.850912 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0b98c1-7867fb5b9f-pnhpm" podStartSLOduration=1.850899038 podStartE2EDuration="1.850899038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:25:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:25:14.849106628 +0000 UTC m=+3996.878588922" watchObservedRunningTime="2026-11-03 02:25:14.850899038 +0000 UTC m=+3996.880381334" Nov 03 02:25:15.458309 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:25:15.458266 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:25:20.201191 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:25:20.201150 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:25:24.989005 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:25:24.988953 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:25:25.740329 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:25:25.740299 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0b98c1-7867fb5b9f-pnhpm" Nov 03 02:25:29.804623 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:25:29.804579 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:25:32.792279 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:25:32.792238 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:25:34.126917 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:25:34.126876 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:25:34.681981 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:25:34.681946 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:25:34.888045 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:25:34.888006 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:25:35.094544 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:25:35.094507 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:25:35.305742 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:25:35.305700 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:25:35.510992 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:25:35.510890 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:25:39.814053 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:25:39.814017 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:25:40.094955 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:25:40.094870 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:25:40.306011 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:25:40.305967 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:25:43.137520 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:25:43.137472 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:25:43.338587 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:25:43.338553 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:25:43.492178 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:25:43.492094 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-b6c56b-759c8fb4d7-54gk6"] Nov 03 02:25:43.496519 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:25:43.496496 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b6c56b-759c8fb4d7-54gk6" Nov 03 02:25:43.552279 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:25:43.552243 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-b6c56b-dockercfg-56xjw\"" Nov 03 02:25:43.613907 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:25:43.604309 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-b6c56b\"" Nov 03 02:25:43.613907 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:25:43.609089 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1c3ee08-9d4b-406f-a183-649ab4a88536-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-b6c56b-759c8fb4d7-54gk6\" (UID: \"f1c3ee08-9d4b-406f-a183-649ab4a88536\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b6c56b-759c8fb4d7-54gk6" Nov 03 02:25:43.613907 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:25:43.609147 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7fzb\" (UniqueName: \"kubernetes.io/projected/f1c3ee08-9d4b-406f-a183-649ab4a88536-kube-api-access-h7fzb\") pod \"payload-pre-processing-e2e-extmod-b-b6c56b-759c8fb4d7-54gk6\" (UID: \"f1c3ee08-9d4b-406f-a183-649ab4a88536\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b6c56b-759c8fb4d7-54gk6" Nov 03 02:25:43.613907 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:25:43.611719 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-b6c56b-759c8fb4d7-54gk6"] Nov 03 02:25:43.658506 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:25:43.658467 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-b6c56b-6f48cf5c6f-5mswj"] Nov 03 02:25:43.662884 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:25:43.662857 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-b6c56b-6f48cf5c6f-5mswj" Nov 03 02:25:43.704268 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:25:43.704235 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-b6c56b-6f48cf5c6f-5mswj"] Nov 03 02:25:43.710749 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:25:43.710718 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1c3ee08-9d4b-406f-a183-649ab4a88536-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-b6c56b-759c8fb4d7-54gk6\" (UID: \"f1c3ee08-9d4b-406f-a183-649ab4a88536\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b6c56b-759c8fb4d7-54gk6" Nov 03 02:25:43.710906 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:25:43.710773 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7fzb\" (UniqueName: \"kubernetes.io/projected/f1c3ee08-9d4b-406f-a183-649ab4a88536-kube-api-access-h7fzb\") pod \"payload-pre-processing-e2e-extmod-b-b6c56b-759c8fb4d7-54gk6\" (UID: \"f1c3ee08-9d4b-406f-a183-649ab4a88536\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b6c56b-759c8fb4d7-54gk6" Nov 03 02:25:43.710979 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:25:43.710917 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdj86\" (UniqueName: \"kubernetes.io/projected/9d61f263-6bc7-40bd-ab1f-f7db9e106c31-kube-api-access-tdj86\") pod \"payload-processing-e2e-extmod-b-b6c56b-6f48cf5c6f-5mswj\" (UID: \"9d61f263-6bc7-40bd-ab1f-f7db9e106c31\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-b6c56b-6f48cf5c6f-5mswj" Nov 03 02:25:43.710979 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:25:43.710950 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d61f263-6bc7-40bd-ab1f-f7db9e106c31-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-b6c56b-6f48cf5c6f-5mswj\" (UID: \"9d61f263-6bc7-40bd-ab1f-f7db9e106c31\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-b6c56b-6f48cf5c6f-5mswj" Nov 03 02:25:43.711526 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:25:43.711497 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1c3ee08-9d4b-406f-a183-649ab4a88536-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-b6c56b-759c8fb4d7-54gk6\" (UID: \"f1c3ee08-9d4b-406f-a183-649ab4a88536\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b6c56b-759c8fb4d7-54gk6" Nov 03 02:25:43.773575 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:25:43.773504 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7fzb\" (UniqueName: \"kubernetes.io/projected/f1c3ee08-9d4b-406f-a183-649ab4a88536-kube-api-access-h7fzb\") pod \"payload-pre-processing-e2e-extmod-b-b6c56b-759c8fb4d7-54gk6\" (UID: \"f1c3ee08-9d4b-406f-a183-649ab4a88536\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b6c56b-759c8fb4d7-54gk6" Nov 03 02:25:43.811275 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:25:43.811241 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b6c56b-759c8fb4d7-54gk6" Nov 03 02:25:43.811854 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:25:43.811830 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdj86\" (UniqueName: \"kubernetes.io/projected/9d61f263-6bc7-40bd-ab1f-f7db9e106c31-kube-api-access-tdj86\") pod \"payload-processing-e2e-extmod-b-b6c56b-6f48cf5c6f-5mswj\" (UID: \"9d61f263-6bc7-40bd-ab1f-f7db9e106c31\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-b6c56b-6f48cf5c6f-5mswj" Nov 03 02:25:43.811912 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:25:43.811870 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d61f263-6bc7-40bd-ab1f-f7db9e106c31-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-b6c56b-6f48cf5c6f-5mswj\" (UID: \"9d61f263-6bc7-40bd-ab1f-f7db9e106c31\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-b6c56b-6f48cf5c6f-5mswj" Nov 03 02:25:43.812926 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:25:43.812906 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d61f263-6bc7-40bd-ab1f-f7db9e106c31-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-b6c56b-6f48cf5c6f-5mswj\" (UID: \"9d61f263-6bc7-40bd-ab1f-f7db9e106c31\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-b6c56b-6f48cf5c6f-5mswj" Nov 03 02:25:43.875542 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:25:43.875472 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdj86\" (UniqueName: \"kubernetes.io/projected/9d61f263-6bc7-40bd-ab1f-f7db9e106c31-kube-api-access-tdj86\") pod \"payload-processing-e2e-extmod-b-b6c56b-6f48cf5c6f-5mswj\" (UID: \"9d61f263-6bc7-40bd-ab1f-f7db9e106c31\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-b6c56b-6f48cf5c6f-5mswj" Nov 03 02:25:43.977195 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:25:43.977157 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-b6c56b-6f48cf5c6f-5mswj" Nov 03 02:25:44.048687 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:25:44.048647 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 02:25:44.183233 ip-10-0-136-16 kubenswrapper[5437]: W1103 02:25:44.183199 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1c3ee08_9d4b_406f_a183_649ab4a88536.slice/crio-3235306881f8582a423a054cc83d0e06c8619a4ce528dae5f423e18afd7ffd7a WatchSource:0}: Error finding container 3235306881f8582a423a054cc83d0e06c8619a4ce528dae5f423e18afd7ffd7a: Status 404 returned error can't find the container with id 3235306881f8582a423a054cc83d0e06c8619a4ce528dae5f423e18afd7ffd7a Nov 03 02:25:44.186617 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:25:44.186591 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-b6c56b-759c8fb4d7-54gk6"] Nov 03 02:25:44.351791 ip-10-0-136-16 kubenswrapper[5437]: W1103 02:25:44.351754 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d61f263_6bc7_40bd_ab1f_f7db9e106c31.slice/crio-53a2a507877c1af26e0c046fa3084c1ee6421d4e4115c96a1c58d56e29d2cf8c WatchSource:0}: Error finding container 53a2a507877c1af26e0c046fa3084c1ee6421d4e4115c96a1c58d56e29d2cf8c: Status 404 returned error can't find the container with id 53a2a507877c1af26e0c046fa3084c1ee6421d4e4115c96a1c58d56e29d2cf8c Nov 03 02:25:44.354784 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:25:44.354758 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-b6c56b-6f48cf5c6f-5mswj"] Nov 03 02:25:44.929752 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:25:44.929718 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b6c56b-759c8fb4d7-54gk6" event={"ID":"f1c3ee08-9d4b-406f-a183-649ab4a88536","Type":"ContainerStarted","Data":"376b348fe47b3d4675d44f615188e4fd96a6f1c9dedb58f846a29a1253075703"} Nov 03 02:25:44.929964 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:25:44.929774 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b6c56b-759c8fb4d7-54gk6" event={"ID":"f1c3ee08-9d4b-406f-a183-649ab4a88536","Type":"ContainerStarted","Data":"3235306881f8582a423a054cc83d0e06c8619a4ce528dae5f423e18afd7ffd7a"} Nov 03 02:25:44.929964 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:25:44.929799 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b6c56b-759c8fb4d7-54gk6" Nov 03 02:25:44.931550 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:25:44.931528 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-b6c56b-6f48cf5c6f-5mswj" event={"ID":"9d61f263-6bc7-40bd-ab1f-f7db9e106c31","Type":"ContainerStarted","Data":"af1bab038f08217d95a863af3a325fbde233047b792ad614252b4c416e89e2d0"} Nov 03 02:25:44.931617 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:25:44.931572 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-b6c56b-6f48cf5c6f-5mswj" Nov 03 02:25:44.931617 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:25:44.931587 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-b6c56b-6f48cf5c6f-5mswj" event={"ID":"9d61f263-6bc7-40bd-ab1f-f7db9e106c31","Type":"ContainerStarted","Data":"53a2a507877c1af26e0c046fa3084c1ee6421d4e4115c96a1c58d56e29d2cf8c"} Nov 03 02:25:44.968700 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:25:44.968667 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:25:45.050348 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:25:45.050257 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b6c56b-759c8fb4d7-54gk6" podStartSLOduration=2.050239077 podStartE2EDuration="2.050239077s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:25:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:25:45.048501728 +0000 UTC m=+4027.077984035" watchObservedRunningTime="2026-11-03 02:25:45.050239077 +0000 UTC m=+4027.079721374" Nov 03 02:25:45.169819 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:25:45.169761 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-b6c56b-6f48cf5c6f-5mswj" podStartSLOduration=2.169746481 podStartE2EDuration="2.169746481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:25:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:25:45.16720192 +0000 UTC m=+4027.196684229" watchObservedRunningTime="2026-11-03 02:25:45.169746481 +0000 UTC m=+4027.199228775" Nov 03 02:25:50.133726 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:25:50.133685 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:25:55.090133 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:25:55.090077 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:25:55.940847 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:25:55.940817 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-b6c56b-6f48cf5c6f-5mswj" Nov 03 02:25:55.941024 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:25:55.940891 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b6c56b-759c8fb4d7-54gk6" Nov 03 02:25:58.739818 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:25:58.739774 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:25:59.842227 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:25:59.842166 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:26:03.691225 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:26:03.691184 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:26:04.587323 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:26:04.587287 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:26:09.389603 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:26:09.389560 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:26:14.035769 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:26:14.035721 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 02:26:14.114770 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:26:14.114727 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:26:18.864755 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:26:18.864669 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:26:23.617814 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:26:23.617777 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:26:28.331078 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:26:28.331036 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:26:33.084308 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:26:33.084271 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:26:35.207685 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:26:35.207649 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:26:35.714684 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:26:35.714645 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:26:36.163513 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:26:36.163478 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-e40047-openshift-default-7797d7b96f-qtjtx"] Nov 03 02:26:36.168039 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:26:36.168016 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e40047-openshift-default-7797d7b96f-qtjtx" Nov 03 02:26:36.229685 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:26:36.229647 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-e40047-openshift-default-dockercfg-hthc6\"" Nov 03 02:26:36.257287 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:26:36.257232 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/953b62df-8d68-4d1e-b0b2-08b87f292998-istio-data\") pod \"e2e-inf-b-e40047-openshift-default-7797d7b96f-qtjtx\" (UID: \"953b62df-8d68-4d1e-b0b2-08b87f292998\") " pod="openshift-ingress/e2e-inf-b-e40047-openshift-default-7797d7b96f-qtjtx" Nov 03 02:26:36.257497 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:26:36.257296 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-px846\" (UniqueName: \"kubernetes.io/projected/953b62df-8d68-4d1e-b0b2-08b87f292998-kube-api-access-px846\") pod \"e2e-inf-b-e40047-openshift-default-7797d7b96f-qtjtx\" (UID: \"953b62df-8d68-4d1e-b0b2-08b87f292998\") " pod="openshift-ingress/e2e-inf-b-e40047-openshift-default-7797d7b96f-qtjtx" Nov 03 02:26:36.257497 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:26:36.257393 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/953b62df-8d68-4d1e-b0b2-08b87f292998-credential-socket\") pod \"e2e-inf-b-e40047-openshift-default-7797d7b96f-qtjtx\" (UID: \"953b62df-8d68-4d1e-b0b2-08b87f292998\") " pod="openshift-ingress/e2e-inf-b-e40047-openshift-default-7797d7b96f-qtjtx" Nov 03 02:26:36.257621 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:26:36.257498 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/953b62df-8d68-4d1e-b0b2-08b87f292998-workload-socket\") pod \"e2e-inf-b-e40047-openshift-default-7797d7b96f-qtjtx\" (UID: \"953b62df-8d68-4d1e-b0b2-08b87f292998\") " pod="openshift-ingress/e2e-inf-b-e40047-openshift-default-7797d7b96f-qtjtx" Nov 03 02:26:36.257683 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:26:36.257617 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/953b62df-8d68-4d1e-b0b2-08b87f292998-istiod-ca-cert\") pod \"e2e-inf-b-e40047-openshift-default-7797d7b96f-qtjtx\" (UID: \"953b62df-8d68-4d1e-b0b2-08b87f292998\") " pod="openshift-ingress/e2e-inf-b-e40047-openshift-default-7797d7b96f-qtjtx" Nov 03 02:26:36.257738 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:26:36.257690 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/953b62df-8d68-4d1e-b0b2-08b87f292998-istio-envoy\") pod \"e2e-inf-b-e40047-openshift-default-7797d7b96f-qtjtx\" (UID: \"953b62df-8d68-4d1e-b0b2-08b87f292998\") " pod="openshift-ingress/e2e-inf-b-e40047-openshift-default-7797d7b96f-qtjtx" Nov 03 02:26:36.257738 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:26:36.257717 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/953b62df-8d68-4d1e-b0b2-08b87f292998-istio-podinfo\") pod \"e2e-inf-b-e40047-openshift-default-7797d7b96f-qtjtx\" (UID: \"953b62df-8d68-4d1e-b0b2-08b87f292998\") " pod="openshift-ingress/e2e-inf-b-e40047-openshift-default-7797d7b96f-qtjtx" Nov 03 02:26:36.257841 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:26:36.257740 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/953b62df-8d68-4d1e-b0b2-08b87f292998-istio-token\") pod \"e2e-inf-b-e40047-openshift-default-7797d7b96f-qtjtx\" (UID: \"953b62df-8d68-4d1e-b0b2-08b87f292998\") " pod="openshift-ingress/e2e-inf-b-e40047-openshift-default-7797d7b96f-qtjtx" Nov 03 02:26:36.257841 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:26:36.257820 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/953b62df-8d68-4d1e-b0b2-08b87f292998-workload-certs\") pod \"e2e-inf-b-e40047-openshift-default-7797d7b96f-qtjtx\" (UID: \"953b62df-8d68-4d1e-b0b2-08b87f292998\") " pod="openshift-ingress/e2e-inf-b-e40047-openshift-default-7797d7b96f-qtjtx" Nov 03 02:26:36.277129 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:26:36.277095 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-e40047-openshift-default-7797d7b96f-qtjtx"] Nov 03 02:26:36.314156 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:26:36.314117 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:26:36.367387 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:26:36.359406 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/953b62df-8d68-4d1e-b0b2-08b87f292998-istio-data\") pod \"e2e-inf-b-e40047-openshift-default-7797d7b96f-qtjtx\" (UID: \"953b62df-8d68-4d1e-b0b2-08b87f292998\") " pod="openshift-ingress/e2e-inf-b-e40047-openshift-default-7797d7b96f-qtjtx" Nov 03 02:26:36.367387 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:26:36.359470 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-px846\" (UniqueName: \"kubernetes.io/projected/953b62df-8d68-4d1e-b0b2-08b87f292998-kube-api-access-px846\") pod \"e2e-inf-b-e40047-openshift-default-7797d7b96f-qtjtx\" (UID: \"953b62df-8d68-4d1e-b0b2-08b87f292998\") " pod="openshift-ingress/e2e-inf-b-e40047-openshift-default-7797d7b96f-qtjtx" Nov 03 02:26:36.367387 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:26:36.359937 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/953b62df-8d68-4d1e-b0b2-08b87f292998-istio-data\") pod \"e2e-inf-b-e40047-openshift-default-7797d7b96f-qtjtx\" (UID: \"953b62df-8d68-4d1e-b0b2-08b87f292998\") " pod="openshift-ingress/e2e-inf-b-e40047-openshift-default-7797d7b96f-qtjtx" Nov 03 02:26:36.367387 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:26:36.360045 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/953b62df-8d68-4d1e-b0b2-08b87f292998-credential-socket\") pod \"e2e-inf-b-e40047-openshift-default-7797d7b96f-qtjtx\" (UID: \"953b62df-8d68-4d1e-b0b2-08b87f292998\") " pod="openshift-ingress/e2e-inf-b-e40047-openshift-default-7797d7b96f-qtjtx" Nov 03 02:26:36.367387 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:26:36.360093 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/953b62df-8d68-4d1e-b0b2-08b87f292998-workload-socket\") pod \"e2e-inf-b-e40047-openshift-default-7797d7b96f-qtjtx\" (UID: \"953b62df-8d68-4d1e-b0b2-08b87f292998\") " pod="openshift-ingress/e2e-inf-b-e40047-openshift-default-7797d7b96f-qtjtx" Nov 03 02:26:36.367387 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:26:36.360677 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/953b62df-8d68-4d1e-b0b2-08b87f292998-credential-socket\") pod \"e2e-inf-b-e40047-openshift-default-7797d7b96f-qtjtx\" (UID: \"953b62df-8d68-4d1e-b0b2-08b87f292998\") " pod="openshift-ingress/e2e-inf-b-e40047-openshift-default-7797d7b96f-qtjtx" Nov 03 02:26:36.367387 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:26:36.361018 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/953b62df-8d68-4d1e-b0b2-08b87f292998-workload-socket\") pod \"e2e-inf-b-e40047-openshift-default-7797d7b96f-qtjtx\" (UID: \"953b62df-8d68-4d1e-b0b2-08b87f292998\") " pod="openshift-ingress/e2e-inf-b-e40047-openshift-default-7797d7b96f-qtjtx" Nov 03 02:26:36.367387 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:26:36.361092 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/953b62df-8d68-4d1e-b0b2-08b87f292998-istiod-ca-cert\") pod \"e2e-inf-b-e40047-openshift-default-7797d7b96f-qtjtx\" (UID: \"953b62df-8d68-4d1e-b0b2-08b87f292998\") " pod="openshift-ingress/e2e-inf-b-e40047-openshift-default-7797d7b96f-qtjtx" Nov 03 02:26:36.367387 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:26:36.361165 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/953b62df-8d68-4d1e-b0b2-08b87f292998-istio-envoy\") pod \"e2e-inf-b-e40047-openshift-default-7797d7b96f-qtjtx\" (UID: \"953b62df-8d68-4d1e-b0b2-08b87f292998\") " pod="openshift-ingress/e2e-inf-b-e40047-openshift-default-7797d7b96f-qtjtx" Nov 03 02:26:36.367387 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:26:36.361190 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/953b62df-8d68-4d1e-b0b2-08b87f292998-istio-podinfo\") pod \"e2e-inf-b-e40047-openshift-default-7797d7b96f-qtjtx\" (UID: \"953b62df-8d68-4d1e-b0b2-08b87f292998\") " pod="openshift-ingress/e2e-inf-b-e40047-openshift-default-7797d7b96f-qtjtx" Nov 03 02:26:36.367387 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:26:36.361213 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/953b62df-8d68-4d1e-b0b2-08b87f292998-istio-token\") pod \"e2e-inf-b-e40047-openshift-default-7797d7b96f-qtjtx\" (UID: \"953b62df-8d68-4d1e-b0b2-08b87f292998\") " pod="openshift-ingress/e2e-inf-b-e40047-openshift-default-7797d7b96f-qtjtx" Nov 03 02:26:36.367387 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:26:36.361293 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/953b62df-8d68-4d1e-b0b2-08b87f292998-workload-certs\") pod \"e2e-inf-b-e40047-openshift-default-7797d7b96f-qtjtx\" (UID: \"953b62df-8d68-4d1e-b0b2-08b87f292998\") " pod="openshift-ingress/e2e-inf-b-e40047-openshift-default-7797d7b96f-qtjtx" Nov 03 02:26:36.367387 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:26:36.363757 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/953b62df-8d68-4d1e-b0b2-08b87f292998-workload-certs\") pod \"e2e-inf-b-e40047-openshift-default-7797d7b96f-qtjtx\" (UID: \"953b62df-8d68-4d1e-b0b2-08b87f292998\") " pod="openshift-ingress/e2e-inf-b-e40047-openshift-default-7797d7b96f-qtjtx" Nov 03 02:26:36.377985 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:26:36.377947 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/953b62df-8d68-4d1e-b0b2-08b87f292998-istio-envoy\") pod \"e2e-inf-b-e40047-openshift-default-7797d7b96f-qtjtx\" (UID: \"953b62df-8d68-4d1e-b0b2-08b87f292998\") " pod="openshift-ingress/e2e-inf-b-e40047-openshift-default-7797d7b96f-qtjtx" Nov 03 02:26:36.378945 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:26:36.378735 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/953b62df-8d68-4d1e-b0b2-08b87f292998-istio-podinfo\") pod \"e2e-inf-b-e40047-openshift-default-7797d7b96f-qtjtx\" (UID: \"953b62df-8d68-4d1e-b0b2-08b87f292998\") " pod="openshift-ingress/e2e-inf-b-e40047-openshift-default-7797d7b96f-qtjtx" Nov 03 02:26:36.379098 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:26:36.378965 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/953b62df-8d68-4d1e-b0b2-08b87f292998-istiod-ca-cert\") pod \"e2e-inf-b-e40047-openshift-default-7797d7b96f-qtjtx\" (UID: \"953b62df-8d68-4d1e-b0b2-08b87f292998\") " pod="openshift-ingress/e2e-inf-b-e40047-openshift-default-7797d7b96f-qtjtx" Nov 03 02:26:36.423858 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:26:36.423765 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-px846\" (UniqueName: \"kubernetes.io/projected/953b62df-8d68-4d1e-b0b2-08b87f292998-kube-api-access-px846\") pod \"e2e-inf-b-e40047-openshift-default-7797d7b96f-qtjtx\" (UID: \"953b62df-8d68-4d1e-b0b2-08b87f292998\") " pod="openshift-ingress/e2e-inf-b-e40047-openshift-default-7797d7b96f-qtjtx" Nov 03 02:26:36.427853 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:26:36.427821 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/953b62df-8d68-4d1e-b0b2-08b87f292998-istio-token\") pod \"e2e-inf-b-e40047-openshift-default-7797d7b96f-qtjtx\" (UID: \"953b62df-8d68-4d1e-b0b2-08b87f292998\") " pod="openshift-ingress/e2e-inf-b-e40047-openshift-default-7797d7b96f-qtjtx" Nov 03 02:26:36.482810 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:26:36.482748 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e40047-openshift-default-7797d7b96f-qtjtx" Nov 03 02:26:36.577289 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:26:36.577249 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:26:36.912674 ip-10-0-136-16 kubenswrapper[5437]: W1103 02:26:36.912643 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod953b62df_8d68_4d1e_b0b2_08b87f292998.slice/crio-b3661056f0f419985185d17ab007e2d5038a38672fecfe7279f0f7b3be7f2f8b WatchSource:0}: Error finding container b3661056f0f419985185d17ab007e2d5038a38672fecfe7279f0f7b3be7f2f8b: Status 404 returned error can't find the container with id b3661056f0f419985185d17ab007e2d5038a38672fecfe7279f0f7b3be7f2f8b Nov 03 02:26:36.915459 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:26:36.915426 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 02:26:36.915571 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:26:36.915490 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 02:26:36.915571 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:26:36.915533 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 02:26:36.916297 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:26:36.916272 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-e40047-openshift-default-7797d7b96f-qtjtx"] Nov 03 02:26:37.269170 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:26:37.269085 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e40047-openshift-default-7797d7b96f-qtjtx" event={"ID":"953b62df-8d68-4d1e-b0b2-08b87f292998","Type":"ContainerStarted","Data":"1755c83f38d36a8a35c469e652a106f39070819328e2d9222883fd42014bf067"} Nov 03 02:26:37.269170 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:26:37.269146 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e40047-openshift-default-7797d7b96f-qtjtx" event={"ID":"953b62df-8d68-4d1e-b0b2-08b87f292998","Type":"ContainerStarted","Data":"b3661056f0f419985185d17ab007e2d5038a38672fecfe7279f0f7b3be7f2f8b"} Nov 03 02:26:37.393448 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:26:37.393398 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-e40047-openshift-default-7797d7b96f-qtjtx" podStartSLOduration=1.393380273 podStartE2EDuration="1.393380273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:26:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:26:37.392145089 +0000 UTC m=+4079.421627383" watchObservedRunningTime="2026-11-03 02:26:37.393380273 +0000 UTC m=+4079.422862572" Nov 03 02:26:37.483553 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:26:37.483518 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-e40047-openshift-default-7797d7b96f-qtjtx" Nov 03 02:26:37.489515 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:26:37.489485 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-e40047-openshift-default-7797d7b96f-qtjtx" Nov 03 02:26:38.210124 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:26:38.210067 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:26:38.275487 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:26:38.275456 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-e40047-openshift-default-7797d7b96f-qtjtx" Nov 03 02:26:38.276545 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:26:38.276526 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-e40047-openshift-default-7797d7b96f-qtjtx" Nov 03 02:26:38.427399 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:26:38.427346 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:26:40.256436 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:26:40.256382 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:26:40.462464 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:26:40.462427 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:26:40.670637 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:26:40.670598 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:26:43.443106 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:26:43.443070 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:26:43.706411 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:26:43.706271 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:26:43.955587 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:26:43.955552 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e40047-5658977b99-28lz4"] Nov 03 02:26:43.959866 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:26:43.959813 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-e40047-5658977b99-28lz4" Nov 03 02:26:44.016602 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:26:44.016573 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e40047-5658977b99-28lz4"] Nov 03 02:26:44.038924 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:26:44.038884 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 02:26:44.063982 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:26:44.063946 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-e40047\"" Nov 03 02:26:44.064148 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:26:44.064075 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-e40047-dockercfg-j4qrg\"" Nov 03 02:26:44.076559 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:26:44.076529 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cbb2\" (UniqueName: \"kubernetes.io/projected/3004112d-86f4-4c2e-9e49-1f8e08c99933-kube-api-access-4cbb2\") pod \"payload-processing-e2e-inf-b-e40047-5658977b99-28lz4\" (UID: \"3004112d-86f4-4c2e-9e49-1f8e08c99933\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e40047-5658977b99-28lz4" Nov 03 02:26:44.076726 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:26:44.076707 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3004112d-86f4-4c2e-9e49-1f8e08c99933-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-e40047-5658977b99-28lz4\" (UID: \"3004112d-86f4-4c2e-9e49-1f8e08c99933\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e40047-5658977b99-28lz4" Nov 03 02:26:44.177683 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:26:44.177636 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cbb2\" (UniqueName: \"kubernetes.io/projected/3004112d-86f4-4c2e-9e49-1f8e08c99933-kube-api-access-4cbb2\") pod \"payload-processing-e2e-inf-b-e40047-5658977b99-28lz4\" (UID: \"3004112d-86f4-4c2e-9e49-1f8e08c99933\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e40047-5658977b99-28lz4" Nov 03 02:26:44.177855 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:26:44.177828 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3004112d-86f4-4c2e-9e49-1f8e08c99933-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-e40047-5658977b99-28lz4\" (UID: \"3004112d-86f4-4c2e-9e49-1f8e08c99933\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e40047-5658977b99-28lz4" Nov 03 02:26:44.178585 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:26:44.178563 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3004112d-86f4-4c2e-9e49-1f8e08c99933-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-e40047-5658977b99-28lz4\" (UID: \"3004112d-86f4-4c2e-9e49-1f8e08c99933\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e40047-5658977b99-28lz4" Nov 03 02:26:44.242409 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:26:44.242302 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cbb2\" (UniqueName: \"kubernetes.io/projected/3004112d-86f4-4c2e-9e49-1f8e08c99933-kube-api-access-4cbb2\") pod \"payload-processing-e2e-inf-b-e40047-5658977b99-28lz4\" (UID: \"3004112d-86f4-4c2e-9e49-1f8e08c99933\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e40047-5658977b99-28lz4" Nov 03 02:26:44.274449 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:26:44.274414 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-e40047-5658977b99-28lz4" Nov 03 02:26:44.648526 ip-10-0-136-16 kubenswrapper[5437]: W1103 02:26:44.648488 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3004112d_86f4_4c2e_9e49_1f8e08c99933.slice/crio-32e73446fef5311dc1afbfad3ddc0ac93c56bff6c360cf742747cc688ae2de72 WatchSource:0}: Error finding container 32e73446fef5311dc1afbfad3ddc0ac93c56bff6c360cf742747cc688ae2de72: Status 404 returned error can't find the container with id 32e73446fef5311dc1afbfad3ddc0ac93c56bff6c360cf742747cc688ae2de72 Nov 03 02:26:44.651697 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:26:44.651668 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e40047-5658977b99-28lz4"] Nov 03 02:26:45.325639 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:26:45.325601 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-e40047-5658977b99-28lz4" event={"ID":"3004112d-86f4-4c2e-9e49-1f8e08c99933","Type":"ContainerStarted","Data":"e4089511b04f82f57fedfaa48981facf928554a2b771c5653101f5312e72f99b"} Nov 03 02:26:45.325835 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:26:45.325658 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-e40047-5658977b99-28lz4" event={"ID":"3004112d-86f4-4c2e-9e49-1f8e08c99933","Type":"ContainerStarted","Data":"32e73446fef5311dc1afbfad3ddc0ac93c56bff6c360cf742747cc688ae2de72"} Nov 03 02:26:45.325835 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:26:45.325695 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-e40047-5658977b99-28lz4" Nov 03 02:26:45.448201 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:26:45.448134 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-e40047-5658977b99-28lz4" podStartSLOduration=2.448114165 podStartE2EDuration="2.448114165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:26:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:26:45.444904912 +0000 UTC m=+4087.474387218" watchObservedRunningTime="2026-11-03 02:26:45.448114165 +0000 UTC m=+4087.477596462" Nov 03 02:26:47.105115 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:26:47.105068 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:26:48.470019 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:26:48.469982 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:26:49.942445 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:26:49.942403 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:26:50.143849 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:26:50.143816 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:26:52.252563 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:26:52.252528 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d6fbd7-849f757bff-r85rr"] Nov 03 02:26:52.575385 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:26:52.575324 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:26:52.810566 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:26:52.810532 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:26:53.212163 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:26:53.212125 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:26:53.535759 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:26:53.535660 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:26:56.335291 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:26:56.335261 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-e40047-5658977b99-28lz4" Nov 03 02:26:57.253250 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:26:57.253203 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d6fbd7-849f757bff-r85rr" podUID="47743531-fbba-44ab-82a9-42b69510b4fe" containerName="payload-pre-processing" containerID="cri-o://08582c8a4986e83c7e6c6fd74c4079cf55afb72cf01273106ca016403471dcc3" gracePeriod=25 Nov 03 02:26:57.417683 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:26:57.417644 5437 generic.go:356] "Generic (PLEG): container finished" podID="47743531-fbba-44ab-82a9-42b69510b4fe" containerID="08582c8a4986e83c7e6c6fd74c4079cf55afb72cf01273106ca016403471dcc3" exitCode=0 Nov 03 02:26:57.418092 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:26:57.417723 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d6fbd7-849f757bff-r85rr" event={"ID":"47743531-fbba-44ab-82a9-42b69510b4fe","Type":"ContainerDied","Data":"08582c8a4986e83c7e6c6fd74c4079cf55afb72cf01273106ca016403471dcc3"} Nov 03 02:26:57.541110 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:26:57.541083 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d6fbd7-849f757bff-r85rr" Nov 03 02:26:57.647749 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:26:57.647703 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47743531-fbba-44ab-82a9-42b69510b4fe-kube-api-access-96tvg\" (UniqueName: \"kubernetes.io/projected/47743531-fbba-44ab-82a9-42b69510b4fe-kube-api-access-96tvg\") pod \"47743531-fbba-44ab-82a9-42b69510b4fe\" (UID: \"47743531-fbba-44ab-82a9-42b69510b4fe\") " Nov 03 02:26:57.647917 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:26:57.647856 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47743531-fbba-44ab-82a9-42b69510b4fe-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47743531-fbba-44ab-82a9-42b69510b4fe-plugins-config-volume\") pod \"47743531-fbba-44ab-82a9-42b69510b4fe\" (UID: \"47743531-fbba-44ab-82a9-42b69510b4fe\") " Nov 03 02:26:57.648310 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:26:57.648275 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47743531-fbba-44ab-82a9-42b69510b4fe-plugins-config-volume" pod "47743531-fbba-44ab-82a9-42b69510b4fe" (UID: "47743531-fbba-44ab-82a9-42b69510b4fe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:26:57.648476 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:26:57.648451 5437 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47743531-fbba-44ab-82a9-42b69510b4fe-plugins-config-volume\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:26:57.650036 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:26:57.650013 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47743531-fbba-44ab-82a9-42b69510b4fe-kube-api-access-96tvg" pod "47743531-fbba-44ab-82a9-42b69510b4fe" (UID: "47743531-fbba-44ab-82a9-42b69510b4fe"). InnerVolumeSpecName "kube-api-access-96tvg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:26:57.750679 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:26:57.750644 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96tvg\" (UniqueName: \"kubernetes.io/projected/47743531-fbba-44ab-82a9-42b69510b4fe-kube-api-access-96tvg\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:26:58.008617 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:26:58.008578 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:26:58.427380 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:26:58.427334 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d6fbd7-849f757bff-r85rr" Nov 03 02:26:58.427380 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:26:58.427343 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d6fbd7-849f757bff-r85rr" event={"ID":"47743531-fbba-44ab-82a9-42b69510b4fe","Type":"ContainerDied","Data":"d90b7c7d42828279a366b80b2a3ef1e5c6b195f220f6e5459a748d50595a06bb"} Nov 03 02:26:58.427868 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:26:58.427429 5437 scope.go:122] "RemoveContainer" containerID="08582c8a4986e83c7e6c6fd74c4079cf55afb72cf01273106ca016403471dcc3" Nov 03 02:26:58.599767 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:26:58.599731 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d6fbd7-849f757bff-r85rr"] Nov 03 02:26:58.650028 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:26:58.649994 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d6fbd7-849f757bff-r85rr"] Nov 03 02:26:59.082645 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:26:59.082608 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:26:59.414744 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:26:59.414650 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:27:00.090025 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:27:00.089991 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47743531-fbba-44ab-82a9-42b69510b4fe" path="/var/lib/kubelet/pods/47743531-fbba-44ab-82a9-42b69510b4fe/volumes" Nov 03 02:27:02.720428 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:27:02.720385 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:27:04.034377 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:27:04.034305 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:27:07.485956 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:27:07.485891 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:27:10.167972 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:27:10.167920 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:27:12.221025 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:27:12.220985 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:27:12.425936 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:27:12.425901 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:27:12.628223 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:27:12.628179 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:27:14.035769 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:27:14.035735 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 02:27:16.027035 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:27:16.026975 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:27:16.281936 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:27:16.281850 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:27:16.486672 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:27:16.486631 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:27:17.111415 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:27:17.111353 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:27:21.863338 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:27:21.863302 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:27:22.078385 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:27:22.078325 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:27:26.601215 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:27:26.601174 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:27:31.327322 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:27:31.327279 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:27:36.064492 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:27:36.064450 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:27:40.779466 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:27:40.779417 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:27:44.035916 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:27:44.035881 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 02:27:45.593062 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:27:45.593026 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:27:50.393339 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:27:50.393241 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:27:55.185860 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:27:55.185819 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:27:59.434028 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:27:59.433983 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:27:59.968162 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:27:59.968123 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:28:04.380342 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:28:04.380305 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:28:04.670452 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:28:04.670301 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:28:07.691313 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:28:07.691276 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:28:09.406949 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:28:09.406913 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:28:13.061349 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:28:13.061308 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0b98c1-7867fb5b9f-pnhpm"] Nov 03 02:28:13.470411 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:28:13.470293 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:28:13.682468 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:28:13.682425 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:28:13.896891 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:28:13.896848 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:28:14.036721 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:28:14.036682 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 02:28:14.752636 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:28:14.752597 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:28:15.131074 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:28:15.131033 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:28:15.381444 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:28:15.381342 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:28:17.100156 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:28:17.100116 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:28:18.062436 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:28:18.062391 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0b98c1-7867fb5b9f-pnhpm" podUID="427bd62b-14cd-49e0-9b4e-4fa30b021ce1" containerName="payload-pre-processing" containerID="cri-o://ed041cb7ca41f75af07e3032017115f2a66e7fa2f072f5899d9a4b1af9f8408f" gracePeriod=25 Nov 03 02:28:18.332148 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:28:18.332079 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0b98c1-7867fb5b9f-pnhpm" Nov 03 02:28:18.468713 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:28:18.468681 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/427bd62b-14cd-49e0-9b4e-4fa30b021ce1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/427bd62b-14cd-49e0-9b4e-4fa30b021ce1-plugins-config-volume\") pod \"427bd62b-14cd-49e0-9b4e-4fa30b021ce1\" (UID: \"427bd62b-14cd-49e0-9b4e-4fa30b021ce1\") " Nov 03 02:28:18.468921 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:28:18.468901 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/427bd62b-14cd-49e0-9b4e-4fa30b021ce1-kube-api-access-vn6cs\" (UniqueName: \"kubernetes.io/projected/427bd62b-14cd-49e0-9b4e-4fa30b021ce1-kube-api-access-vn6cs\") pod \"427bd62b-14cd-49e0-9b4e-4fa30b021ce1\" (UID: \"427bd62b-14cd-49e0-9b4e-4fa30b021ce1\") " Nov 03 02:28:18.469152 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:28:18.469124 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/427bd62b-14cd-49e0-9b4e-4fa30b021ce1-plugins-config-volume" pod "427bd62b-14cd-49e0-9b4e-4fa30b021ce1" (UID: "427bd62b-14cd-49e0-9b4e-4fa30b021ce1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:28:18.469548 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:28:18.469521 5437 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/427bd62b-14cd-49e0-9b4e-4fa30b021ce1-plugins-config-volume\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:28:18.471058 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:28:18.471023 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/427bd62b-14cd-49e0-9b4e-4fa30b021ce1-kube-api-access-vn6cs" pod "427bd62b-14cd-49e0-9b4e-4fa30b021ce1" (UID: "427bd62b-14cd-49e0-9b4e-4fa30b021ce1"). InnerVolumeSpecName "kube-api-access-vn6cs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:28:18.570707 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:28:18.570674 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vn6cs\" (UniqueName: \"kubernetes.io/projected/427bd62b-14cd-49e0-9b4e-4fa30b021ce1-kube-api-access-vn6cs\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:28:18.990675 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:28:18.990639 5437 generic.go:356] "Generic (PLEG): container finished" podID="427bd62b-14cd-49e0-9b4e-4fa30b021ce1" containerID="ed041cb7ca41f75af07e3032017115f2a66e7fa2f072f5899d9a4b1af9f8408f" exitCode=0 Nov 03 02:28:18.990861 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:28:18.990684 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0b98c1-7867fb5b9f-pnhpm" event={"ID":"427bd62b-14cd-49e0-9b4e-4fa30b021ce1","Type":"ContainerDied","Data":"ed041cb7ca41f75af07e3032017115f2a66e7fa2f072f5899d9a4b1af9f8408f"} Nov 03 02:28:18.990861 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:28:18.990710 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0b98c1-7867fb5b9f-pnhpm" Nov 03 02:28:18.990861 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:28:18.990749 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0b98c1-7867fb5b9f-pnhpm" event={"ID":"427bd62b-14cd-49e0-9b4e-4fa30b021ce1","Type":"ContainerDied","Data":"4e56ad5cc63c404443e646fae5056d4e4c2822e8e09381beee4138aeae713b4c"} Nov 03 02:28:18.990861 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:28:18.990772 5437 scope.go:122] "RemoveContainer" containerID="ed041cb7ca41f75af07e3032017115f2a66e7fa2f072f5899d9a4b1af9f8408f" Nov 03 02:28:19.001589 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:28:19.001568 5437 scope.go:122] "RemoveContainer" containerID="ed041cb7ca41f75af07e3032017115f2a66e7fa2f072f5899d9a4b1af9f8408f" Nov 03 02:28:19.001914 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:28:19.001892 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed041cb7ca41f75af07e3032017115f2a66e7fa2f072f5899d9a4b1af9f8408f\": container with ID starting with ed041cb7ca41f75af07e3032017115f2a66e7fa2f072f5899d9a4b1af9f8408f not found: ID does not exist" containerID="ed041cb7ca41f75af07e3032017115f2a66e7fa2f072f5899d9a4b1af9f8408f" Nov 03 02:28:19.001992 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:28:19.001925 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed041cb7ca41f75af07e3032017115f2a66e7fa2f072f5899d9a4b1af9f8408f"} err="failed to get container status \"ed041cb7ca41f75af07e3032017115f2a66e7fa2f072f5899d9a4b1af9f8408f\": rpc error: code = NotFound desc = could not find container \"ed041cb7ca41f75af07e3032017115f2a66e7fa2f072f5899d9a4b1af9f8408f\": container with ID starting with ed041cb7ca41f75af07e3032017115f2a66e7fa2f072f5899d9a4b1af9f8408f not found: ID does not exist" Nov 03 02:28:19.161163 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:28:19.161127 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0b98c1-7867fb5b9f-pnhpm"] Nov 03 02:28:19.212721 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:28:19.212685 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0b98c1-7867fb5b9f-pnhpm"] Nov 03 02:28:19.774981 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:28:19.774946 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:28:20.091383 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:28:20.091337 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="427bd62b-14cd-49e0-9b4e-4fa30b021ce1" path="/var/lib/kubelet/pods/427bd62b-14cd-49e0-9b4e-4fa30b021ce1/volumes" Nov 03 02:28:21.928782 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:28:21.928741 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:28:24.604726 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:28:24.604687 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:28:24.807613 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:28:24.807571 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:28:25.475902 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:28:25.475863 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:28:30.517098 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:28:30.517046 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:28:35.439107 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:28:35.439063 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:28:38.906577 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:28:38.906537 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:28:39.317529 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:28:39.317491 5437 patch_prober.go:29] interesting pod/e2e-inf-b-e40047-openshift-default-7797d7b96f-qtjtx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 02:28:39.317725 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:28:39.317581 5437 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-e40047-openshift-default-7797d7b96f-qtjtx" podUID="953b62df-8d68-4d1e-b0b2-08b87f292998" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 02:28:39.590949 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:28:39.590808 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:28:41.138043 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:28:41.138006 5437 generic.go:356] "Generic (PLEG): container finished" podID="953b62df-8d68-4d1e-b0b2-08b87f292998" containerID="1755c83f38d36a8a35c469e652a106f39070819328e2d9222883fd42014bf067" exitCode=0 Nov 03 02:28:41.138544 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:28:41.138079 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e40047-openshift-default-7797d7b96f-qtjtx" event={"ID":"953b62df-8d68-4d1e-b0b2-08b87f292998","Type":"ContainerDied","Data":"1755c83f38d36a8a35c469e652a106f39070819328e2d9222883fd42014bf067"} Nov 03 02:28:41.138544 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:28:41.138522 5437 scope.go:122] "RemoveContainer" containerID="1755c83f38d36a8a35c469e652a106f39070819328e2d9222883fd42014bf067" Nov 03 02:28:41.140325 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:28:41.140298 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 02:28:41.140409 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:28:41.140384 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 02:28:41.140453 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:28:41.140419 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 02:28:41.297343 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:28:41.297251 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:28:41.726480 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:28:41.726397 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:28:42.150542 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:28:42.150486 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e40047-openshift-default-7797d7b96f-qtjtx" event={"ID":"953b62df-8d68-4d1e-b0b2-08b87f292998","Type":"ContainerStarted","Data":"6210fe4358a0b67625a90cf5ae7e8b8e82bedb784b282a235424b8e6ea6267eb"} Nov 03 02:28:42.245782 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:28:42.245741 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:28:42.483209 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:28:42.483121 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-e40047-openshift-default-7797d7b96f-qtjtx" Nov 03 02:28:42.483419 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:28:42.483396 5437 patch_prober.go:29] interesting pod/e2e-inf-b-e40047-openshift-default-7797d7b96f-qtjtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body= Nov 03 02:28:42.483490 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:28:42.483444 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e40047-openshift-default-7797d7b96f-qtjtx" podUID="953b62df-8d68-4d1e-b0b2-08b87f292998" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" Nov 03 02:28:43.484176 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:28:43.484137 5437 patch_prober.go:29] interesting pod/e2e-inf-b-e40047-openshift-default-7797d7b96f-qtjtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body= Nov 03 02:28:43.484627 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:28:43.484196 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e40047-openshift-default-7797d7b96f-qtjtx" podUID="953b62df-8d68-4d1e-b0b2-08b87f292998" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" Nov 03 02:28:44.056902 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:28:44.056860 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 02:28:44.167472 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:28:44.167435 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-ea5c75-569b6776c9-pqphg"] Nov 03 02:28:44.272047 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:28:44.272011 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-ea5c75-67b8d7c68c-vv2np"] Nov 03 02:28:44.483594 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:28:44.483516 5437 patch_prober.go:29] interesting pod/e2e-inf-b-e40047-openshift-default-7797d7b96f-qtjtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body= Nov 03 02:28:44.483594 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:28:44.483579 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e40047-openshift-default-7797d7b96f-qtjtx" podUID="953b62df-8d68-4d1e-b0b2-08b87f292998" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" Nov 03 02:28:44.583943 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:28:44.583906 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:28:44.783955 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:28:44.783817 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:28:44.999638 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:28:44.999599 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:28:45.374824 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:28:45.374794 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:28:45.484235 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:28:45.484191 5437 patch_prober.go:29] interesting pod/e2e-inf-b-e40047-openshift-default-7797d7b96f-qtjtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body= Nov 03 02:28:45.484417 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:28:45.484274 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e40047-openshift-default-7797d7b96f-qtjtx" podUID="953b62df-8d68-4d1e-b0b2-08b87f292998" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" Nov 03 02:28:46.050720 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:28:46.050680 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:28:46.369472 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:28:46.369387 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:28:46.483862 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:28:46.483802 5437 patch_prober.go:29] interesting pod/e2e-inf-b-e40047-openshift-default-7797d7b96f-qtjtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body= Nov 03 02:28:46.484046 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:28:46.483882 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e40047-openshift-default-7797d7b96f-qtjtx" podUID="953b62df-8d68-4d1e-b0b2-08b87f292998" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" Nov 03 02:28:46.572083 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:28:46.572020 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:28:47.483721 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:28:47.483683 5437 patch_prober.go:29] interesting pod/e2e-inf-b-e40047-openshift-default-7797d7b96f-qtjtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body= Nov 03 02:28:47.484156 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:28:47.483737 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e40047-openshift-default-7797d7b96f-qtjtx" podUID="953b62df-8d68-4d1e-b0b2-08b87f292998" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" Nov 03 02:28:48.487820 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:28:48.487787 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-e40047-openshift-default-7797d7b96f-qtjtx" Nov 03 02:28:48.488307 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:28:48.488285 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-e40047-openshift-default-7797d7b96f-qtjtx" Nov 03 02:28:48.489116 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:28:48.489097 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-e40047-openshift-default-7797d7b96f-qtjtx" Nov 03 02:28:49.167759 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:28:49.167715 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-ea5c75-569b6776c9-pqphg" podUID="85d7dce8-f705-45e7-a5a9-6a5ff7ef1f2d" containerName="payload-processing" containerID="cri-o://b66ea622c395d947328eeb56fc21088876da60cf7536b48f87e8b36a30b36881" gracePeriod=25 Nov 03 02:28:49.273424 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:28:49.273357 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ea5c75-67b8d7c68c-vv2np" podUID="a753f6ce-4198-4196-96de-73ff6c81b32a" containerName="payload-pre-processing" containerID="cri-o://b2e0c7216f1cb0690209780cdc138eb315b59a7d1da4f1d0d73040441c65c2e6" gracePeriod=25 Nov 03 02:28:49.453201 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:28:49.453173 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-ea5c75-569b6776c9-pqphg" Nov 03 02:28:49.525741 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:28:49.525703 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85d7dce8-f705-45e7-a5a9-6a5ff7ef1f2d-kube-api-access-6m6p4\" (UniqueName: \"kubernetes.io/projected/85d7dce8-f705-45e7-a5a9-6a5ff7ef1f2d-kube-api-access-6m6p4\") pod \"85d7dce8-f705-45e7-a5a9-6a5ff7ef1f2d\" (UID: \"85d7dce8-f705-45e7-a5a9-6a5ff7ef1f2d\") " Nov 03 02:28:49.526211 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:28:49.525989 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85d7dce8-f705-45e7-a5a9-6a5ff7ef1f2d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85d7dce8-f705-45e7-a5a9-6a5ff7ef1f2d-plugins-config-volume\") pod \"85d7dce8-f705-45e7-a5a9-6a5ff7ef1f2d\" (UID: \"85d7dce8-f705-45e7-a5a9-6a5ff7ef1f2d\") " Nov 03 02:28:49.526352 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:28:49.526329 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85d7dce8-f705-45e7-a5a9-6a5ff7ef1f2d-plugins-config-volume" pod "85d7dce8-f705-45e7-a5a9-6a5ff7ef1f2d" (UID: "85d7dce8-f705-45e7-a5a9-6a5ff7ef1f2d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:28:49.526587 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:28:49.526568 5437 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85d7dce8-f705-45e7-a5a9-6a5ff7ef1f2d-plugins-config-volume\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:28:49.527992 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:28:49.527972 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85d7dce8-f705-45e7-a5a9-6a5ff7ef1f2d-kube-api-access-6m6p4" pod "85d7dce8-f705-45e7-a5a9-6a5ff7ef1f2d" (UID: "85d7dce8-f705-45e7-a5a9-6a5ff7ef1f2d"). InnerVolumeSpecName "kube-api-access-6m6p4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:28:49.549227 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:28:49.549202 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ea5c75-67b8d7c68c-vv2np" Nov 03 02:28:49.627264 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:28:49.627227 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a753f6ce-4198-4196-96de-73ff6c81b32a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a753f6ce-4198-4196-96de-73ff6c81b32a-plugins-config-volume\") pod \"a753f6ce-4198-4196-96de-73ff6c81b32a\" (UID: \"a753f6ce-4198-4196-96de-73ff6c81b32a\") " Nov 03 02:28:49.627463 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:28:49.627276 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a753f6ce-4198-4196-96de-73ff6c81b32a-kube-api-access-k5ncz\" (UniqueName: \"kubernetes.io/projected/a753f6ce-4198-4196-96de-73ff6c81b32a-kube-api-access-k5ncz\") pod \"a753f6ce-4198-4196-96de-73ff6c81b32a\" (UID: \"a753f6ce-4198-4196-96de-73ff6c81b32a\") " Nov 03 02:28:49.627713 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:28:49.627686 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a753f6ce-4198-4196-96de-73ff6c81b32a-plugins-config-volume" pod "a753f6ce-4198-4196-96de-73ff6c81b32a" (UID: "a753f6ce-4198-4196-96de-73ff6c81b32a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:28:49.627837 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:28:49.627819 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6m6p4\" (UniqueName: \"kubernetes.io/projected/85d7dce8-f705-45e7-a5a9-6a5ff7ef1f2d-kube-api-access-6m6p4\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:28:49.627999 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:28:49.627839 5437 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a753f6ce-4198-4196-96de-73ff6c81b32a-plugins-config-volume\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:28:49.629577 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:28:49.629551 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a753f6ce-4198-4196-96de-73ff6c81b32a-kube-api-access-k5ncz" pod "a753f6ce-4198-4196-96de-73ff6c81b32a" (UID: "a753f6ce-4198-4196-96de-73ff6c81b32a"). InnerVolumeSpecName "kube-api-access-k5ncz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:28:49.728912 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:28:49.728830 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5ncz\" (UniqueName: \"kubernetes.io/projected/a753f6ce-4198-4196-96de-73ff6c81b32a-kube-api-access-k5ncz\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:28:50.202902 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:28:50.202863 5437 generic.go:356] "Generic (PLEG): container finished" podID="85d7dce8-f705-45e7-a5a9-6a5ff7ef1f2d" containerID="b66ea622c395d947328eeb56fc21088876da60cf7536b48f87e8b36a30b36881" exitCode=0 Nov 03 02:28:50.203091 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:28:50.202910 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-ea5c75-569b6776c9-pqphg" Nov 03 02:28:50.203091 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:28:50.202991 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-ea5c75-569b6776c9-pqphg" event={"ID":"85d7dce8-f705-45e7-a5a9-6a5ff7ef1f2d","Type":"ContainerDied","Data":"b66ea622c395d947328eeb56fc21088876da60cf7536b48f87e8b36a30b36881"} Nov 03 02:28:50.203091 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:28:50.203040 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-ea5c75-569b6776c9-pqphg" event={"ID":"85d7dce8-f705-45e7-a5a9-6a5ff7ef1f2d","Type":"ContainerDied","Data":"13d43a766d85587810fbb5b846c13403da3938043e357f6878a5e914f2ba58e9"} Nov 03 02:28:50.203091 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:28:50.203071 5437 scope.go:122] "RemoveContainer" containerID="b66ea622c395d947328eeb56fc21088876da60cf7536b48f87e8b36a30b36881" Nov 03 02:28:50.204525 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:28:50.204504 5437 generic.go:356] "Generic (PLEG): container finished" podID="a753f6ce-4198-4196-96de-73ff6c81b32a" containerID="b2e0c7216f1cb0690209780cdc138eb315b59a7d1da4f1d0d73040441c65c2e6" exitCode=0 Nov 03 02:28:50.204633 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:28:50.204606 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ea5c75-67b8d7c68c-vv2np" Nov 03 02:28:50.204711 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:28:50.204645 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ea5c75-67b8d7c68c-vv2np" event={"ID":"a753f6ce-4198-4196-96de-73ff6c81b32a","Type":"ContainerDied","Data":"b2e0c7216f1cb0690209780cdc138eb315b59a7d1da4f1d0d73040441c65c2e6"} Nov 03 02:28:50.204711 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:28:50.204688 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ea5c75-67b8d7c68c-vv2np" event={"ID":"a753f6ce-4198-4196-96de-73ff6c81b32a","Type":"ContainerDied","Data":"72fc3a6f25a8706cae6e1a3b3e7efdcbeaf4fda22b6e525e477ea032f8839808"} Nov 03 02:28:50.214310 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:28:50.214293 5437 scope.go:122] "RemoveContainer" containerID="b66ea622c395d947328eeb56fc21088876da60cf7536b48f87e8b36a30b36881" Nov 03 02:28:50.214627 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:28:50.214597 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b66ea622c395d947328eeb56fc21088876da60cf7536b48f87e8b36a30b36881\": container with ID starting with b66ea622c395d947328eeb56fc21088876da60cf7536b48f87e8b36a30b36881 not found: ID does not exist" containerID="b66ea622c395d947328eeb56fc21088876da60cf7536b48f87e8b36a30b36881" Nov 03 02:28:50.214683 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:28:50.214643 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b66ea622c395d947328eeb56fc21088876da60cf7536b48f87e8b36a30b36881"} err="failed to get container status \"b66ea622c395d947328eeb56fc21088876da60cf7536b48f87e8b36a30b36881\": rpc error: code = NotFound desc = could not find container \"b66ea622c395d947328eeb56fc21088876da60cf7536b48f87e8b36a30b36881\": container with ID starting with b66ea622c395d947328eeb56fc21088876da60cf7536b48f87e8b36a30b36881 not found: ID does not exist" Nov 03 02:28:50.214683 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:28:50.214667 5437 scope.go:122] "RemoveContainer" containerID="b2e0c7216f1cb0690209780cdc138eb315b59a7d1da4f1d0d73040441c65c2e6" Nov 03 02:28:50.223986 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:28:50.223965 5437 scope.go:122] "RemoveContainer" containerID="b2e0c7216f1cb0690209780cdc138eb315b59a7d1da4f1d0d73040441c65c2e6" Nov 03 02:28:50.224259 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:28:50.224240 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2e0c7216f1cb0690209780cdc138eb315b59a7d1da4f1d0d73040441c65c2e6\": container with ID starting with b2e0c7216f1cb0690209780cdc138eb315b59a7d1da4f1d0d73040441c65c2e6 not found: ID does not exist" containerID="b2e0c7216f1cb0690209780cdc138eb315b59a7d1da4f1d0d73040441c65c2e6" Nov 03 02:28:50.224310 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:28:50.224267 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2e0c7216f1cb0690209780cdc138eb315b59a7d1da4f1d0d73040441c65c2e6"} err="failed to get container status \"b2e0c7216f1cb0690209780cdc138eb315b59a7d1da4f1d0d73040441c65c2e6\": rpc error: code = NotFound desc = could not find container \"b2e0c7216f1cb0690209780cdc138eb315b59a7d1da4f1d0d73040441c65c2e6\": container with ID starting with b2e0c7216f1cb0690209780cdc138eb315b59a7d1da4f1d0d73040441c65c2e6 not found: ID does not exist" Nov 03 02:28:50.372875 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:28:50.372785 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-ea5c75-569b6776c9-pqphg"] Nov 03 02:28:50.424465 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:28:50.424431 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-ea5c75-569b6776c9-pqphg"] Nov 03 02:28:50.491534 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:28:50.491493 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-ea5c75-67b8d7c68c-vv2np"] Nov 03 02:28:50.495187 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:28:50.495162 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-ea5c75-67b8d7c68c-vv2np"] Nov 03 02:28:51.153525 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:28:51.153490 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:28:51.737119 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:28:51.737078 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:28:52.089120 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:28:52.089084 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85d7dce8-f705-45e7-a5a9-6a5ff7ef1f2d" path="/var/lib/kubelet/pods/85d7dce8-f705-45e7-a5a9-6a5ff7ef1f2d/volumes" Nov 03 02:28:52.089611 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:28:52.089595 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a753f6ce-4198-4196-96de-73ff6c81b32a" path="/var/lib/kubelet/pods/a753f6ce-4198-4196-96de-73ff6c81b32a/volumes" Nov 03 02:28:56.736490 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:28:56.736440 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:28:59.781769 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:28:59.781730 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:29:00.036621 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:29:00.036532 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:29:01.071340 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:29:01.071300 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:29:01.683537 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:29:01.683484 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:29:04.749011 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:29:04.748961 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:29:06.483928 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:29:06.483880 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:29:07.039765 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:29:07.039722 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:29:11.312321 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:29:11.312284 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:29:14.036398 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:29:14.036334 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 02:29:16.067944 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:29:16.067900 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:29:20.783708 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:29:20.783614 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:29:22.710077 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:29:22.710042 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:29:22.916246 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:29:22.916211 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:29:25.493258 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:29:25.493220 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:29:30.205606 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:29:30.205563 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:29:35.150202 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:29:35.150166 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:29:39.279621 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:29:39.279586 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:29:39.739760 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:29:39.739721 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:29:44.035605 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:29:44.035565 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 02:29:44.536825 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:29:44.536785 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:29:45.555994 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:29:45.555954 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:29:45.758407 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:29:45.758326 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:29:45.977708 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:29:45.977610 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:29:49.005630 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:29:49.005584 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:29:49.704933 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:29:49.704901 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:29:50.622814 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:29:50.622766 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:29:50.876097 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:29:50.876000 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:29:51.103996 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:29:51.103953 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:29:51.744268 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:29:51.744228 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:29:52.099590 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:29:52.099546 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:29:52.364695 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:29:52.364599 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:29:53.443996 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:29:53.443952 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:29:53.641745 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:29:53.641702 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:29:54.523170 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:29:54.523121 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:29:54.740475 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:29:54.740439 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:29:54.950109 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:29:54.950069 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:29:55.153402 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:29:55.153349 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:29:55.856228 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:29:55.856195 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:29:56.495423 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:29:56.495361 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:29:59.909980 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:29:59.909931 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-ea5c75-5858f8984d-7xnbm"] Nov 03 02:29:59.910418 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:29:59.910172 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ea5c75-5858f8984d-7xnbm" podUID="a22e3efc-1ac5-454a-9af2-7526a058bac6" containerName="maas-api" containerID="cri-o://d6c09a018a8ee13e3e5d79bba06cb2107251ad27618eb493cd41f5afd2bc1624" gracePeriod=30 Nov 03 02:30:00.070136 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:00.070098 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:30:00.185075 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:00.184997 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ea5c75-5858f8984d-7xnbm" Nov 03 02:30:00.226589 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:00.226549 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a22e3efc-1ac5-454a-9af2-7526a058bac6-kube-api-access-nwqmd\" (UniqueName: \"kubernetes.io/projected/a22e3efc-1ac5-454a-9af2-7526a058bac6-kube-api-access-nwqmd\") pod \"a22e3efc-1ac5-454a-9af2-7526a058bac6\" (UID: \"a22e3efc-1ac5-454a-9af2-7526a058bac6\") " Nov 03 02:30:00.226793 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:00.226736 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a22e3efc-1ac5-454a-9af2-7526a058bac6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a22e3efc-1ac5-454a-9af2-7526a058bac6-metrics-certs\") pod \"a22e3efc-1ac5-454a-9af2-7526a058bac6\" (UID: \"a22e3efc-1ac5-454a-9af2-7526a058bac6\") " Nov 03 02:30:00.226863 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:00.226802 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a22e3efc-1ac5-454a-9af2-7526a058bac6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a22e3efc-1ac5-454a-9af2-7526a058bac6-maas-api-tls\") pod \"a22e3efc-1ac5-454a-9af2-7526a058bac6\" (UID: \"a22e3efc-1ac5-454a-9af2-7526a058bac6\") " Nov 03 02:30:00.226863 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:00.226836 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a22e3efc-1ac5-454a-9af2-7526a058bac6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a22e3efc-1ac5-454a-9af2-7526a058bac6-openshift-service-ca\") pod \"a22e3efc-1ac5-454a-9af2-7526a058bac6\" (UID: \"a22e3efc-1ac5-454a-9af2-7526a058bac6\") " Nov 03 02:30:00.227262 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:00.227236 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a22e3efc-1ac5-454a-9af2-7526a058bac6-openshift-service-ca" pod "a22e3efc-1ac5-454a-9af2-7526a058bac6" (UID: "a22e3efc-1ac5-454a-9af2-7526a058bac6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:30:00.228789 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:00.228760 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a22e3efc-1ac5-454a-9af2-7526a058bac6-kube-api-access-nwqmd" pod "a22e3efc-1ac5-454a-9af2-7526a058bac6" (UID: "a22e3efc-1ac5-454a-9af2-7526a058bac6"). InnerVolumeSpecName "kube-api-access-nwqmd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:30:00.228915 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:00.228830 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a22e3efc-1ac5-454a-9af2-7526a058bac6-metrics-certs" pod "a22e3efc-1ac5-454a-9af2-7526a058bac6" (UID: "a22e3efc-1ac5-454a-9af2-7526a058bac6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:30:00.228915 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:00.228850 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a22e3efc-1ac5-454a-9af2-7526a058bac6-maas-api-tls" pod "a22e3efc-1ac5-454a-9af2-7526a058bac6" (UID: "a22e3efc-1ac5-454a-9af2-7526a058bac6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:30:00.293939 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:00.293901 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894550-fdxp4"] Nov 03 02:30:00.294546 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:00.294523 5437 cpu_manager.go:414] "containerMap: removing container" podUID="85d7dce8-f705-45e7-a5a9-6a5ff7ef1f2d" containerName="payload-processing" Nov 03 02:30:00.294546 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:00.294542 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85d7dce8-f705-45e7-a5a9-6a5ff7ef1f2d" containerName="payload-processing" Nov 03 02:30:00.294770 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:00.294572 5437 cpu_manager.go:414] "containerMap: removing container" podUID="427bd62b-14cd-49e0-9b4e-4fa30b021ce1" containerName="payload-pre-processing" Nov 03 02:30:00.294770 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:00.294579 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="427bd62b-14cd-49e0-9b4e-4fa30b021ce1" containerName="payload-pre-processing" Nov 03 02:30:00.294770 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:00.294591 5437 cpu_manager.go:414] "containerMap: removing container" podUID="47743531-fbba-44ab-82a9-42b69510b4fe" containerName="payload-pre-processing" Nov 03 02:30:00.294770 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:00.294597 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47743531-fbba-44ab-82a9-42b69510b4fe" containerName="payload-pre-processing" Nov 03 02:30:00.294770 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:00.294612 5437 cpu_manager.go:414] "containerMap: removing container" podUID="a753f6ce-4198-4196-96de-73ff6c81b32a" containerName="payload-pre-processing" Nov 03 02:30:00.294770 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:00.294620 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a753f6ce-4198-4196-96de-73ff6c81b32a" containerName="payload-pre-processing" Nov 03 02:30:00.294770 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:00.294648 5437 cpu_manager.go:414] "containerMap: removing container" podUID="a22e3efc-1ac5-454a-9af2-7526a058bac6" containerName="maas-api" Nov 03 02:30:00.294770 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:00.294656 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a22e3efc-1ac5-454a-9af2-7526a058bac6" containerName="maas-api" Nov 03 02:30:00.295274 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:00.294835 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="a22e3efc-1ac5-454a-9af2-7526a058bac6" containerName="maas-api" Nov 03 02:30:00.295274 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:00.294856 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="85d7dce8-f705-45e7-a5a9-6a5ff7ef1f2d" containerName="payload-processing" Nov 03 02:30:00.295274 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:00.294868 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="427bd62b-14cd-49e0-9b4e-4fa30b021ce1" containerName="payload-pre-processing" Nov 03 02:30:00.295274 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:00.294877 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="47743531-fbba-44ab-82a9-42b69510b4fe" containerName="payload-pre-processing" Nov 03 02:30:00.295274 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:00.294887 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="a753f6ce-4198-4196-96de-73ff6c81b32a" containerName="payload-pre-processing" Nov 03 02:30:00.297919 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:00.297899 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894550-fdxp4" Nov 03 02:30:00.305831 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:00.305801 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1c9b4e-29894550-wm9w5"] Nov 03 02:30:00.309453 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:00.309433 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1c9b4e-29894550-wm9w5" Nov 03 02:30:00.328562 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:00.328506 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpnqq\" (UniqueName: \"kubernetes.io/projected/e52dc6a2-2ad8-4e48-89b2-94530d8b248f-kube-api-access-kpnqq\") pod \"maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894550-fdxp4\" (UID: \"e52dc6a2-2ad8-4e48-89b2-94530d8b248f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894550-fdxp4" Nov 03 02:30:00.329205 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:00.328639 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be4f3bdc-1caa-4c1a-abb6-4bef99793882-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-1c9b4e-29894550-wm9w5\" (UID: \"be4f3bdc-1caa-4c1a-abb6-4bef99793882\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1c9b4e-29894550-wm9w5" Nov 03 02:30:00.329205 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:00.328678 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhfg8\" (UniqueName: \"kubernetes.io/projected/be4f3bdc-1caa-4c1a-abb6-4bef99793882-kube-api-access-lhfg8\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-1c9b4e-29894550-wm9w5\" (UID: \"be4f3bdc-1caa-4c1a-abb6-4bef99793882\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1c9b4e-29894550-wm9w5" Nov 03 02:30:00.329205 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:00.328712 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e52dc6a2-2ad8-4e48-89b2-94530d8b248f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894550-fdxp4\" (UID: \"e52dc6a2-2ad8-4e48-89b2-94530d8b248f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894550-fdxp4" Nov 03 02:30:00.329205 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:00.328857 5437 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a22e3efc-1ac5-454a-9af2-7526a058bac6-metrics-certs\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:30:00.329205 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:00.328872 5437 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a22e3efc-1ac5-454a-9af2-7526a058bac6-maas-api-tls\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:30:00.329205 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:00.328888 5437 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a22e3efc-1ac5-454a-9af2-7526a058bac6-openshift-service-ca\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:30:00.329205 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:00.328900 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwqmd\" (UniqueName: \"kubernetes.io/projected/a22e3efc-1ac5-454a-9af2-7526a058bac6-kube-api-access-nwqmd\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:30:00.336579 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:00.336554 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41aa5f-29894550-kfrjc"] Nov 03 02:30:00.342106 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:00.341635 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894550-tx85q"] Nov 03 02:30:00.343283 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:00.343257 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41aa5f-29894550-kfrjc" Nov 03 02:30:00.347299 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:00.347126 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894550-tx85q" Nov 03 02:30:00.361884 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:00.361288 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-434b6a-29894550-ltk6s"] Nov 03 02:30:00.372026 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:00.370893 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-434b6a-29894550-ltk6s" Nov 03 02:30:00.372026 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:00.371701 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894550-fdxp4"] Nov 03 02:30:00.377635 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:00.377545 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1c9b4e-29894550-wm9w5"] Nov 03 02:30:00.380229 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:00.379694 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:30:00.405789 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:00.405753 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894550-tx85q"] Nov 03 02:30:00.409984 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:00.409949 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41aa5f-29894550-kfrjc"] Nov 03 02:30:00.417552 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:00.417520 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-434b6a-29894550-ltk6s"] Nov 03 02:30:00.429812 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:00.429782 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ab7c82f-91ef-4685-99bb-cdb46cd0f1b1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-434b6a-29894550-ltk6s\" (UID: \"1ab7c82f-91ef-4685-99bb-cdb46cd0f1b1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-434b6a-29894550-ltk6s" Nov 03 02:30:00.430075 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:00.429821 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98c0de5e-7908-451f-aa0d-fc9e732a4ed7-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894550-tx85q\" (UID: \"98c0de5e-7908-451f-aa0d-fc9e732a4ed7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894550-tx85q" Nov 03 02:30:00.430075 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:00.429931 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpnqq\" (UniqueName: \"kubernetes.io/projected/e52dc6a2-2ad8-4e48-89b2-94530d8b248f-kube-api-access-kpnqq\") pod \"maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894550-fdxp4\" (UID: \"e52dc6a2-2ad8-4e48-89b2-94530d8b248f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894550-fdxp4" Nov 03 02:30:00.430075 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:00.430046 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be4f3bdc-1caa-4c1a-abb6-4bef99793882-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-1c9b4e-29894550-wm9w5\" (UID: \"be4f3bdc-1caa-4c1a-abb6-4bef99793882\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1c9b4e-29894550-wm9w5" Nov 03 02:30:00.431055 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:00.430081 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fgwsl\"" Nov 03 02:30:00.431055 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:00.430079 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhfg8\" (UniqueName: \"kubernetes.io/projected/be4f3bdc-1caa-4c1a-abb6-4bef99793882-kube-api-access-lhfg8\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-1c9b4e-29894550-wm9w5\" (UID: \"be4f3bdc-1caa-4c1a-abb6-4bef99793882\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1c9b4e-29894550-wm9w5" Nov 03 02:30:00.431055 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:00.430129 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e52dc6a2-2ad8-4e48-89b2-94530d8b248f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894550-fdxp4\" (UID: \"e52dc6a2-2ad8-4e48-89b2-94530d8b248f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894550-fdxp4" Nov 03 02:30:00.431055 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:00.430148 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5308b4cd-02d4-4c3b-8b64-72063794535f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-41aa5f-29894550-kfrjc\" (UID: \"5308b4cd-02d4-4c3b-8b64-72063794535f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41aa5f-29894550-kfrjc" Nov 03 02:30:00.431055 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:00.430188 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cwt4\" (UniqueName: \"kubernetes.io/projected/5308b4cd-02d4-4c3b-8b64-72063794535f-kube-api-access-4cwt4\") pod \"maas-api-key-cleanup-e2e-worker-w3-41aa5f-29894550-kfrjc\" (UID: \"5308b4cd-02d4-4c3b-8b64-72063794535f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41aa5f-29894550-kfrjc" Nov 03 02:30:00.431055 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:00.430323 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ggf9\" (UniqueName: \"kubernetes.io/projected/98c0de5e-7908-451f-aa0d-fc9e732a4ed7-kube-api-access-5ggf9\") pod \"maas-api-key-cleanup-29894550-tx85q\" (UID: \"98c0de5e-7908-451f-aa0d-fc9e732a4ed7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894550-tx85q" Nov 03 02:30:00.431055 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:00.430407 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndw26\" (UniqueName: \"kubernetes.io/projected/1ab7c82f-91ef-4685-99bb-cdb46cd0f1b1-kube-api-access-ndw26\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-434b6a-29894550-ltk6s\" (UID: \"1ab7c82f-91ef-4685-99bb-cdb46cd0f1b1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-434b6a-29894550-ltk6s" Nov 03 02:30:00.431533 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:00.431169 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e52dc6a2-2ad8-4e48-89b2-94530d8b248f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894550-fdxp4\" (UID: \"e52dc6a2-2ad8-4e48-89b2-94530d8b248f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894550-fdxp4" Nov 03 02:30:00.431533 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:00.431211 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be4f3bdc-1caa-4c1a-abb6-4bef99793882-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-1c9b4e-29894550-wm9w5\" (UID: \"be4f3bdc-1caa-4c1a-abb6-4bef99793882\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1c9b4e-29894550-wm9w5" Nov 03 02:30:00.493820 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:00.493738 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpnqq\" (UniqueName: \"kubernetes.io/projected/e52dc6a2-2ad8-4e48-89b2-94530d8b248f-kube-api-access-kpnqq\") pod \"maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894550-fdxp4\" (UID: \"e52dc6a2-2ad8-4e48-89b2-94530d8b248f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894550-fdxp4" Nov 03 02:30:00.493820 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:00.493738 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhfg8\" (UniqueName: \"kubernetes.io/projected/be4f3bdc-1caa-4c1a-abb6-4bef99793882-kube-api-access-lhfg8\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-1c9b4e-29894550-wm9w5\" (UID: \"be4f3bdc-1caa-4c1a-abb6-4bef99793882\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1c9b4e-29894550-wm9w5" Nov 03 02:30:00.531854 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:00.531810 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndw26\" (UniqueName: \"kubernetes.io/projected/1ab7c82f-91ef-4685-99bb-cdb46cd0f1b1-kube-api-access-ndw26\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-434b6a-29894550-ltk6s\" (UID: \"1ab7c82f-91ef-4685-99bb-cdb46cd0f1b1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-434b6a-29894550-ltk6s" Nov 03 02:30:00.532053 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:00.531910 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ab7c82f-91ef-4685-99bb-cdb46cd0f1b1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-434b6a-29894550-ltk6s\" (UID: \"1ab7c82f-91ef-4685-99bb-cdb46cd0f1b1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-434b6a-29894550-ltk6s" Nov 03 02:30:00.532053 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:00.531946 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98c0de5e-7908-451f-aa0d-fc9e732a4ed7-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894550-tx85q\" (UID: \"98c0de5e-7908-451f-aa0d-fc9e732a4ed7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894550-tx85q" Nov 03 02:30:00.532180 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:00.532049 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5308b4cd-02d4-4c3b-8b64-72063794535f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-41aa5f-29894550-kfrjc\" (UID: \"5308b4cd-02d4-4c3b-8b64-72063794535f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41aa5f-29894550-kfrjc" Nov 03 02:30:00.532180 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:00.532102 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cwt4\" (UniqueName: \"kubernetes.io/projected/5308b4cd-02d4-4c3b-8b64-72063794535f-kube-api-access-4cwt4\") pod \"maas-api-key-cleanup-e2e-worker-w3-41aa5f-29894550-kfrjc\" (UID: \"5308b4cd-02d4-4c3b-8b64-72063794535f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41aa5f-29894550-kfrjc" Nov 03 02:30:00.532288 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:00.532210 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ggf9\" (UniqueName: \"kubernetes.io/projected/98c0de5e-7908-451f-aa0d-fc9e732a4ed7-kube-api-access-5ggf9\") pod \"maas-api-key-cleanup-29894550-tx85q\" (UID: \"98c0de5e-7908-451f-aa0d-fc9e732a4ed7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894550-tx85q" Nov 03 02:30:00.532723 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:00.532698 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ab7c82f-91ef-4685-99bb-cdb46cd0f1b1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-434b6a-29894550-ltk6s\" (UID: \"1ab7c82f-91ef-4685-99bb-cdb46cd0f1b1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-434b6a-29894550-ltk6s" Nov 03 02:30:00.532841 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:00.532820 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5308b4cd-02d4-4c3b-8b64-72063794535f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-41aa5f-29894550-kfrjc\" (UID: \"5308b4cd-02d4-4c3b-8b64-72063794535f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41aa5f-29894550-kfrjc" Nov 03 02:30:00.532885 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:00.532842 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98c0de5e-7908-451f-aa0d-fc9e732a4ed7-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894550-tx85q\" (UID: \"98c0de5e-7908-451f-aa0d-fc9e732a4ed7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894550-tx85q" Nov 03 02:30:00.596077 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:00.596038 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndw26\" (UniqueName: \"kubernetes.io/projected/1ab7c82f-91ef-4685-99bb-cdb46cd0f1b1-kube-api-access-ndw26\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-434b6a-29894550-ltk6s\" (UID: \"1ab7c82f-91ef-4685-99bb-cdb46cd0f1b1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-434b6a-29894550-ltk6s" Nov 03 02:30:00.596077 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:00.596065 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cwt4\" (UniqueName: \"kubernetes.io/projected/5308b4cd-02d4-4c3b-8b64-72063794535f-kube-api-access-4cwt4\") pod \"maas-api-key-cleanup-e2e-worker-w3-41aa5f-29894550-kfrjc\" (UID: \"5308b4cd-02d4-4c3b-8b64-72063794535f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41aa5f-29894550-kfrjc" Nov 03 02:30:00.596280 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:00.596099 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ggf9\" (UniqueName: \"kubernetes.io/projected/98c0de5e-7908-451f-aa0d-fc9e732a4ed7-kube-api-access-5ggf9\") pod \"maas-api-key-cleanup-29894550-tx85q\" (UID: \"98c0de5e-7908-451f-aa0d-fc9e732a4ed7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894550-tx85q" Nov 03 02:30:00.612230 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:00.612202 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894550-fdxp4" Nov 03 02:30:00.624696 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:00.624673 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1c9b4e-29894550-wm9w5" Nov 03 02:30:00.672854 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:00.672665 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41aa5f-29894550-kfrjc" Nov 03 02:30:00.678249 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:00.678217 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894550-tx85q" Nov 03 02:30:00.701985 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:00.701953 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-434b6a-29894550-ltk6s" Nov 03 02:30:00.772432 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:00.772007 5437 generic.go:356] "Generic (PLEG): container finished" podID="a22e3efc-1ac5-454a-9af2-7526a058bac6" containerID="d6c09a018a8ee13e3e5d79bba06cb2107251ad27618eb493cd41f5afd2bc1624" exitCode=0 Nov 03 02:30:00.772432 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:00.772210 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ea5c75-5858f8984d-7xnbm" event={"ID":"a22e3efc-1ac5-454a-9af2-7526a058bac6","Type":"ContainerDied","Data":"d6c09a018a8ee13e3e5d79bba06cb2107251ad27618eb493cd41f5afd2bc1624"} Nov 03 02:30:00.772432 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:00.772277 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ea5c75-5858f8984d-7xnbm" event={"ID":"a22e3efc-1ac5-454a-9af2-7526a058bac6","Type":"ContainerDied","Data":"e9eae570f969fe1dfc25bb9013a62e8443b8d32e56802c1eb9de1896d74b7f30"} Nov 03 02:30:00.772432 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:00.772305 5437 scope.go:122] "RemoveContainer" containerID="d6c09a018a8ee13e3e5d79bba06cb2107251ad27618eb493cd41f5afd2bc1624" Nov 03 02:30:00.773416 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:00.772809 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ea5c75-5858f8984d-7xnbm" Nov 03 02:30:00.806020 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:00.804936 5437 scope.go:122] "RemoveContainer" containerID="d6c09a018a8ee13e3e5d79bba06cb2107251ad27618eb493cd41f5afd2bc1624" Nov 03 02:30:00.810087 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:30:00.810053 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6c09a018a8ee13e3e5d79bba06cb2107251ad27618eb493cd41f5afd2bc1624\": container with ID starting with d6c09a018a8ee13e3e5d79bba06cb2107251ad27618eb493cd41f5afd2bc1624 not found: ID does not exist" containerID="d6c09a018a8ee13e3e5d79bba06cb2107251ad27618eb493cd41f5afd2bc1624" Nov 03 02:30:00.810202 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:00.810099 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6c09a018a8ee13e3e5d79bba06cb2107251ad27618eb493cd41f5afd2bc1624"} err="failed to get container status \"d6c09a018a8ee13e3e5d79bba06cb2107251ad27618eb493cd41f5afd2bc1624\": rpc error: code = NotFound desc = could not find container \"d6c09a018a8ee13e3e5d79bba06cb2107251ad27618eb493cd41f5afd2bc1624\": container with ID starting with d6c09a018a8ee13e3e5d79bba06cb2107251ad27618eb493cd41f5afd2bc1624 not found: ID does not exist" Nov 03 02:30:00.995101 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:00.995067 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894550-fdxp4"] Nov 03 02:30:00.998891 ip-10-0-136-16 kubenswrapper[5437]: W1103 02:30:00.998857 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode52dc6a2_2ad8_4e48_89b2_94530d8b248f.slice/crio-9224e75ee66e1f04addc5ea1ec2be4b15d8c1fa928109a5faa5ccfa07e941733 WatchSource:0}: Error finding container 9224e75ee66e1f04addc5ea1ec2be4b15d8c1fa928109a5faa5ccfa07e941733: Status 404 returned error can't find the container with id 9224e75ee66e1f04addc5ea1ec2be4b15d8c1fa928109a5faa5ccfa07e941733 Nov 03 02:30:01.040516 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:01.040485 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1c9b4e-29894550-wm9w5"] Nov 03 02:30:01.045608 ip-10-0-136-16 kubenswrapper[5437]: W1103 02:30:01.045548 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe4f3bdc_1caa_4c1a_abb6_4bef99793882.slice/crio-f5ae21a4e829cf9e686bfd9859dd2664297b2ed345cb6aa16fe5374d3f5ee909 WatchSource:0}: Error finding container f5ae21a4e829cf9e686bfd9859dd2664297b2ed345cb6aa16fe5374d3f5ee909: Status 404 returned error can't find the container with id f5ae21a4e829cf9e686bfd9859dd2664297b2ed345cb6aa16fe5374d3f5ee909 Nov 03 02:30:01.048311 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:01.048292 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-ea5c75-5858f8984d-7xnbm"] Nov 03 02:30:01.050886 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:01.050866 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-ea5c75-5858f8984d-7xnbm"] Nov 03 02:30:01.096592 ip-10-0-136-16 kubenswrapper[5437]: W1103 02:30:01.096564 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5308b4cd_02d4_4c3b_8b64_72063794535f.slice/crio-60ed9e097ca921dbf33a458e1ef30e50a8d23a9498dc26e01374aac95c085d13 WatchSource:0}: Error finding container 60ed9e097ca921dbf33a458e1ef30e50a8d23a9498dc26e01374aac95c085d13: Status 404 returned error can't find the container with id 60ed9e097ca921dbf33a458e1ef30e50a8d23a9498dc26e01374aac95c085d13 Nov 03 02:30:01.100895 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:01.100869 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41aa5f-29894550-kfrjc"] Nov 03 02:30:01.101736 ip-10-0-136-16 kubenswrapper[5437]: W1103 02:30:01.101709 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98c0de5e_7908_451f_aa0d_fc9e732a4ed7.slice/crio-692179fdc21205e8c0477731d56216bbf8b4c4ee9e251dcf1375a26eb3360dfc WatchSource:0}: Error finding container 692179fdc21205e8c0477731d56216bbf8b4c4ee9e251dcf1375a26eb3360dfc: Status 404 returned error can't find the container with id 692179fdc21205e8c0477731d56216bbf8b4c4ee9e251dcf1375a26eb3360dfc Nov 03 02:30:01.106901 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:01.106872 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894550-tx85q"] Nov 03 02:30:01.144395 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:01.144353 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-434b6a-29894550-ltk6s"] Nov 03 02:30:01.222410 ip-10-0-136-16 kubenswrapper[5437]: W1103 02:30:01.221931 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ab7c82f_91ef_4685_99bb_cdb46cd0f1b1.slice/crio-3f22d176989a9b8fe7df540dc92072b6c111303d2025960c810241a65a3bb85a WatchSource:0}: Error finding container 3f22d176989a9b8fe7df540dc92072b6c111303d2025960c810241a65a3bb85a: Status 404 returned error can't find the container with id 3f22d176989a9b8fe7df540dc92072b6c111303d2025960c810241a65a3bb85a Nov 03 02:30:01.780845 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:01.780806 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-434b6a-29894550-ltk6s" event={"ID":"1ab7c82f-91ef-4685-99bb-cdb46cd0f1b1","Type":"ContainerStarted","Data":"3f22d176989a9b8fe7df540dc92072b6c111303d2025960c810241a65a3bb85a"} Nov 03 02:30:01.782467 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:01.782438 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1c9b4e-29894550-wm9w5" event={"ID":"be4f3bdc-1caa-4c1a-abb6-4bef99793882","Type":"ContainerStarted","Data":"b812936b9f743d416d1e6e55b508969c86385cc7d72b67da775689a5556ea958"} Nov 03 02:30:01.782602 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:01.782482 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1c9b4e-29894550-wm9w5" event={"ID":"be4f3bdc-1caa-4c1a-abb6-4bef99793882","Type":"ContainerStarted","Data":"f5ae21a4e829cf9e686bfd9859dd2664297b2ed345cb6aa16fe5374d3f5ee909"} Nov 03 02:30:01.784050 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:01.784018 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894550-fdxp4" event={"ID":"e52dc6a2-2ad8-4e48-89b2-94530d8b248f","Type":"ContainerStarted","Data":"9bfd7d16b50daf01cefc6e33e60fa48399adc91fd53ca4613ad9cf83e0af3e31"} Nov 03 02:30:01.784183 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:01.784062 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894550-fdxp4" event={"ID":"e52dc6a2-2ad8-4e48-89b2-94530d8b248f","Type":"ContainerStarted","Data":"9224e75ee66e1f04addc5ea1ec2be4b15d8c1fa928109a5faa5ccfa07e941733"} Nov 03 02:30:01.785507 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:01.785482 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894550-tx85q" event={"ID":"98c0de5e-7908-451f-aa0d-fc9e732a4ed7","Type":"ContainerStarted","Data":"0689d249c27ae7c04b1e56e690576816100d21a6b6c533a5e192d35a0b424034"} Nov 03 02:30:01.785612 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:01.785525 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894550-tx85q" event={"ID":"98c0de5e-7908-451f-aa0d-fc9e732a4ed7","Type":"ContainerStarted","Data":"692179fdc21205e8c0477731d56216bbf8b4c4ee9e251dcf1375a26eb3360dfc"} Nov 03 02:30:01.786985 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:01.786956 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41aa5f-29894550-kfrjc" event={"ID":"5308b4cd-02d4-4c3b-8b64-72063794535f","Type":"ContainerStarted","Data":"f6fa3bd08a17c9eb2d27afe0c9f66862d022bfcdfee35ad49b7e7ab5dbfa4dcd"} Nov 03 02:30:01.787078 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:01.787008 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41aa5f-29894550-kfrjc" event={"ID":"5308b4cd-02d4-4c3b-8b64-72063794535f","Type":"ContainerStarted","Data":"60ed9e097ca921dbf33a458e1ef30e50a8d23a9498dc26e01374aac95c085d13"} Nov 03 02:30:01.885104 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:01.885018 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:30:01.900742 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:01.900690 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1c9b4e-29894550-wm9w5" podStartSLOduration=1.900677687 podStartE2EDuration="1.900677687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:30:01.900495631 +0000 UTC m=+4283.929977926" watchObservedRunningTime="2026-11-03 02:30:01.900677687 +0000 UTC m=+4283.930159979" Nov 03 02:30:02.021760 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:02.021578 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894550-tx85q" podStartSLOduration=2.021560426 podStartE2EDuration="2.021560426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:30:02.01865274 +0000 UTC m=+4284.048135038" watchObservedRunningTime="2026-11-03 02:30:02.021560426 +0000 UTC m=+4284.051042724" Nov 03 02:30:02.089595 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:02.089561 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a22e3efc-1ac5-454a-9af2-7526a058bac6" path="/var/lib/kubelet/pods/a22e3efc-1ac5-454a-9af2-7526a058bac6/volumes" Nov 03 02:30:02.143102 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:02.142990 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894550-fdxp4" podStartSLOduration=2.142975518 podStartE2EDuration="2.142975518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:30:02.140156451 +0000 UTC m=+4284.169638746" watchObservedRunningTime="2026-11-03 02:30:02.142975518 +0000 UTC m=+4284.172457813" Nov 03 02:30:02.263456 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:02.263396 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41aa5f-29894550-kfrjc" podStartSLOduration=2.263360436 podStartE2EDuration="2.263360436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:30:02.259878354 +0000 UTC m=+4284.289360648" watchObservedRunningTime="2026-11-03 02:30:02.263360436 +0000 UTC m=+4284.292842732" Nov 03 02:30:02.796229 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:02.796136 5437 generic.go:356] "Generic (PLEG): container finished" podID="5308b4cd-02d4-4c3b-8b64-72063794535f" containerID="f6fa3bd08a17c9eb2d27afe0c9f66862d022bfcdfee35ad49b7e7ab5dbfa4dcd" exitCode=60 Nov 03 02:30:02.796412 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:02.796225 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41aa5f-29894550-kfrjc" event={"ID":"5308b4cd-02d4-4c3b-8b64-72063794535f","Type":"ContainerDied","Data":"f6fa3bd08a17c9eb2d27afe0c9f66862d022bfcdfee35ad49b7e7ab5dbfa4dcd"} Nov 03 02:30:02.796576 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:02.796555 5437 scope.go:122] "RemoveContainer" containerID="f6fa3bd08a17c9eb2d27afe0c9f66862d022bfcdfee35ad49b7e7ab5dbfa4dcd" Nov 03 02:30:02.798011 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:02.797988 5437 generic.go:356] "Generic (PLEG): container finished" podID="1ab7c82f-91ef-4685-99bb-cdb46cd0f1b1" containerID="4949a62ceaaf9c309e2ff96748e9e671b449115972a5c2a3b6e439fd9133c21e" exitCode=60 Nov 03 02:30:02.798129 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:02.798062 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-434b6a-29894550-ltk6s" event={"ID":"1ab7c82f-91ef-4685-99bb-cdb46cd0f1b1","Type":"ContainerDied","Data":"4949a62ceaaf9c309e2ff96748e9e671b449115972a5c2a3b6e439fd9133c21e"} Nov 03 02:30:02.798328 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:02.798296 5437 scope.go:122] "RemoveContainer" containerID="4949a62ceaaf9c309e2ff96748e9e671b449115972a5c2a3b6e439fd9133c21e" Nov 03 02:30:02.799795 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:02.799768 5437 generic.go:356] "Generic (PLEG): container finished" podID="be4f3bdc-1caa-4c1a-abb6-4bef99793882" containerID="b812936b9f743d416d1e6e55b508969c86385cc7d72b67da775689a5556ea958" exitCode=60 Nov 03 02:30:02.799906 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:02.799881 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1c9b4e-29894550-wm9w5" event={"ID":"be4f3bdc-1caa-4c1a-abb6-4bef99793882","Type":"ContainerDied","Data":"b812936b9f743d416d1e6e55b508969c86385cc7d72b67da775689a5556ea958"} Nov 03 02:30:02.800165 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:02.800057 5437 scope.go:122] "RemoveContainer" containerID="b812936b9f743d416d1e6e55b508969c86385cc7d72b67da775689a5556ea958" Nov 03 02:30:02.801678 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:02.801660 5437 generic.go:356] "Generic (PLEG): container finished" podID="e52dc6a2-2ad8-4e48-89b2-94530d8b248f" containerID="9bfd7d16b50daf01cefc6e33e60fa48399adc91fd53ca4613ad9cf83e0af3e31" exitCode=60 Nov 03 02:30:02.801757 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:02.801690 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894550-fdxp4" event={"ID":"e52dc6a2-2ad8-4e48-89b2-94530d8b248f","Type":"ContainerDied","Data":"9bfd7d16b50daf01cefc6e33e60fa48399adc91fd53ca4613ad9cf83e0af3e31"} Nov 03 02:30:02.801986 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:02.801967 5437 scope.go:122] "RemoveContainer" containerID="9bfd7d16b50daf01cefc6e33e60fa48399adc91fd53ca4613ad9cf83e0af3e31" Nov 03 02:30:02.803265 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:02.803245 5437 generic.go:356] "Generic (PLEG): container finished" podID="98c0de5e-7908-451f-aa0d-fc9e732a4ed7" containerID="0689d249c27ae7c04b1e56e690576816100d21a6b6c533a5e192d35a0b424034" exitCode=60 Nov 03 02:30:02.803351 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:02.803331 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894550-tx85q" event={"ID":"98c0de5e-7908-451f-aa0d-fc9e732a4ed7","Type":"ContainerDied","Data":"0689d249c27ae7c04b1e56e690576816100d21a6b6c533a5e192d35a0b424034"} Nov 03 02:30:02.803492 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:02.803479 5437 scope.go:122] "RemoveContainer" containerID="0689d249c27ae7c04b1e56e690576816100d21a6b6c533a5e192d35a0b424034" Nov 03 02:30:03.812271 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:03.812234 5437 generic.go:356] "Generic (PLEG): container finished" podID="98c0de5e-7908-451f-aa0d-fc9e732a4ed7" containerID="42bccab2b976f7f0c85e2a1ce79cbb64b32daa57b49fd99b4e7dea7c449c960a" exitCode=60 Nov 03 02:30:03.812717 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:03.812319 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894550-tx85q" event={"ID":"98c0de5e-7908-451f-aa0d-fc9e732a4ed7","Type":"ContainerDied","Data":"42bccab2b976f7f0c85e2a1ce79cbb64b32daa57b49fd99b4e7dea7c449c960a"} Nov 03 02:30:03.812717 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:03.812434 5437 scope.go:122] "RemoveContainer" containerID="0689d249c27ae7c04b1e56e690576816100d21a6b6c533a5e192d35a0b424034" Nov 03 02:30:03.812717 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:03.812705 5437 scope.go:122] "RemoveContainer" containerID="42bccab2b976f7f0c85e2a1ce79cbb64b32daa57b49fd99b4e7dea7c449c960a" Nov 03 02:30:03.813052 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:30:03.813028 5437 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894550-tx85q_odh-ai-gateway-infra(98c0de5e-7908-451f-aa0d-fc9e732a4ed7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894550-tx85q" podUID="98c0de5e-7908-451f-aa0d-fc9e732a4ed7" Nov 03 02:30:03.814760 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:03.814736 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41aa5f-29894550-kfrjc" event={"ID":"5308b4cd-02d4-4c3b-8b64-72063794535f","Type":"ContainerStarted","Data":"f8fee504625aeddc3a99d090153134e2073acf7738279fc77ec829c3eb5955c6"} Nov 03 02:30:03.817164 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:03.817143 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-434b6a-29894550-ltk6s" event={"ID":"1ab7c82f-91ef-4685-99bb-cdb46cd0f1b1","Type":"ContainerStarted","Data":"9a812bcaa90d3cda93472114ba731f907d245c93e196d457cf96e086a363a473"} Nov 03 02:30:03.819345 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:03.819322 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1c9b4e-29894550-wm9w5" event={"ID":"be4f3bdc-1caa-4c1a-abb6-4bef99793882","Type":"ContainerStarted","Data":"85f1b44dab5c21f384226df96e83884cfcecb59741bbfca29babea7e957edaa6"} Nov 03 02:30:03.821058 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:03.821037 5437 generic.go:356] "Generic (PLEG): container finished" podID="e52dc6a2-2ad8-4e48-89b2-94530d8b248f" containerID="e70e37d2f3554d102ed3e27c7b7e9f65b27e632ccaaa78819802d599862670a7" exitCode=60 Nov 03 02:30:03.821157 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:03.821076 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894550-fdxp4" event={"ID":"e52dc6a2-2ad8-4e48-89b2-94530d8b248f","Type":"ContainerDied","Data":"e70e37d2f3554d102ed3e27c7b7e9f65b27e632ccaaa78819802d599862670a7"} Nov 03 02:30:03.821402 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:03.821379 5437 scope.go:122] "RemoveContainer" containerID="e70e37d2f3554d102ed3e27c7b7e9f65b27e632ccaaa78819802d599862670a7" Nov 03 02:30:03.821709 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:30:03.821686 5437 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894550-fdxp4_odh-ai-gateway-infra(e52dc6a2-2ad8-4e48-89b2-94530d8b248f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894550-fdxp4" podUID="e52dc6a2-2ad8-4e48-89b2-94530d8b248f" Nov 03 02:30:03.916740 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:03.916716 5437 scope.go:122] "RemoveContainer" containerID="9bfd7d16b50daf01cefc6e33e60fa48399adc91fd53ca4613ad9cf83e0af3e31" Nov 03 02:30:04.169845 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:04.169782 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-434b6a-29894550-ltk6s" podStartSLOduration=4.16976808 podStartE2EDuration="4.16976808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:30:04.168688594 +0000 UTC m=+4286.198170889" watchObservedRunningTime="2026-11-03 02:30:04.16976808 +0000 UTC m=+4286.199250376" Nov 03 02:30:04.829993 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:04.829958 5437 generic.go:356] "Generic (PLEG): container finished" podID="1ab7c82f-91ef-4685-99bb-cdb46cd0f1b1" containerID="9a812bcaa90d3cda93472114ba731f907d245c93e196d457cf96e086a363a473" exitCode=60 Nov 03 02:30:04.830498 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:04.830038 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-434b6a-29894550-ltk6s" event={"ID":"1ab7c82f-91ef-4685-99bb-cdb46cd0f1b1","Type":"ContainerDied","Data":"9a812bcaa90d3cda93472114ba731f907d245c93e196d457cf96e086a363a473"} Nov 03 02:30:04.830498 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:04.830101 5437 scope.go:122] "RemoveContainer" containerID="4949a62ceaaf9c309e2ff96748e9e671b449115972a5c2a3b6e439fd9133c21e" Nov 03 02:30:04.830498 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:04.830331 5437 scope.go:122] "RemoveContainer" containerID="9a812bcaa90d3cda93472114ba731f907d245c93e196d457cf96e086a363a473" Nov 03 02:30:04.830744 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:30:04.830714 5437 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-434b6a-29894550-ltk6s_odh-ai-gateway-infra(1ab7c82f-91ef-4685-99bb-cdb46cd0f1b1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-434b6a-29894550-ltk6s" podUID="1ab7c82f-91ef-4685-99bb-cdb46cd0f1b1" Nov 03 02:30:04.832173 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:04.832149 5437 generic.go:356] "Generic (PLEG): container finished" podID="be4f3bdc-1caa-4c1a-abb6-4bef99793882" containerID="85f1b44dab5c21f384226df96e83884cfcecb59741bbfca29babea7e957edaa6" exitCode=60 Nov 03 02:30:04.832276 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:04.832226 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1c9b4e-29894550-wm9w5" event={"ID":"be4f3bdc-1caa-4c1a-abb6-4bef99793882","Type":"ContainerDied","Data":"85f1b44dab5c21f384226df96e83884cfcecb59741bbfca29babea7e957edaa6"} Nov 03 02:30:04.832556 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:04.832537 5437 scope.go:122] "RemoveContainer" containerID="85f1b44dab5c21f384226df96e83884cfcecb59741bbfca29babea7e957edaa6" Nov 03 02:30:04.832895 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:30:04.832810 5437 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-1c9b4e-29894550-wm9w5_odh-ai-gateway-infra(be4f3bdc-1caa-4c1a-abb6-4bef99793882)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1c9b4e-29894550-wm9w5" podUID="be4f3bdc-1caa-4c1a-abb6-4bef99793882" Nov 03 02:30:04.834177 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:04.834158 5437 scope.go:122] "RemoveContainer" containerID="e70e37d2f3554d102ed3e27c7b7e9f65b27e632ccaaa78819802d599862670a7" Nov 03 02:30:04.834549 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:30:04.834523 5437 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894550-fdxp4_odh-ai-gateway-infra(e52dc6a2-2ad8-4e48-89b2-94530d8b248f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894550-fdxp4" podUID="e52dc6a2-2ad8-4e48-89b2-94530d8b248f" Nov 03 02:30:04.835921 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:04.835905 5437 scope.go:122] "RemoveContainer" containerID="42bccab2b976f7f0c85e2a1ce79cbb64b32daa57b49fd99b4e7dea7c449c960a" Nov 03 02:30:04.836152 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:30:04.836127 5437 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894550-tx85q_odh-ai-gateway-infra(98c0de5e-7908-451f-aa0d-fc9e732a4ed7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894550-tx85q" podUID="98c0de5e-7908-451f-aa0d-fc9e732a4ed7" Nov 03 02:30:04.837827 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:04.837804 5437 generic.go:356] "Generic (PLEG): container finished" podID="5308b4cd-02d4-4c3b-8b64-72063794535f" containerID="f8fee504625aeddc3a99d090153134e2073acf7738279fc77ec829c3eb5955c6" exitCode=60 Nov 03 02:30:04.837925 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:04.837889 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41aa5f-29894550-kfrjc" event={"ID":"5308b4cd-02d4-4c3b-8b64-72063794535f","Type":"ContainerDied","Data":"f8fee504625aeddc3a99d090153134e2073acf7738279fc77ec829c3eb5955c6"} Nov 03 02:30:04.838076 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:04.838062 5437 scope.go:122] "RemoveContainer" containerID="f8fee504625aeddc3a99d090153134e2073acf7738279fc77ec829c3eb5955c6" Nov 03 02:30:04.838268 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:30:04.838248 5437 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-41aa5f-29894550-kfrjc_odh-ai-gateway-infra(5308b4cd-02d4-4c3b-8b64-72063794535f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41aa5f-29894550-kfrjc" podUID="5308b4cd-02d4-4c3b-8b64-72063794535f" Nov 03 02:30:04.845560 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:04.844727 5437 scope.go:122] "RemoveContainer" containerID="b812936b9f743d416d1e6e55b508969c86385cc7d72b67da775689a5556ea958" Nov 03 02:30:04.857745 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:04.857727 5437 scope.go:122] "RemoveContainer" containerID="f6fa3bd08a17c9eb2d27afe0c9f66862d022bfcdfee35ad49b7e7ab5dbfa4dcd" Nov 03 02:30:05.094514 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:05.094421 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:30:05.846738 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:05.846704 5437 scope.go:122] "RemoveContainer" containerID="f8fee504625aeddc3a99d090153134e2073acf7738279fc77ec829c3eb5955c6" Nov 03 02:30:05.847140 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:30:05.847017 5437 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-41aa5f-29894550-kfrjc_odh-ai-gateway-infra(5308b4cd-02d4-4c3b-8b64-72063794535f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41aa5f-29894550-kfrjc" podUID="5308b4cd-02d4-4c3b-8b64-72063794535f" Nov 03 02:30:05.849320 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:05.849296 5437 scope.go:122] "RemoveContainer" containerID="9a812bcaa90d3cda93472114ba731f907d245c93e196d457cf96e086a363a473" Nov 03 02:30:05.849649 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:30:05.849624 5437 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-434b6a-29894550-ltk6s_odh-ai-gateway-infra(1ab7c82f-91ef-4685-99bb-cdb46cd0f1b1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-434b6a-29894550-ltk6s" podUID="1ab7c82f-91ef-4685-99bb-cdb46cd0f1b1" Nov 03 02:30:05.850835 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:05.850818 5437 scope.go:122] "RemoveContainer" containerID="85f1b44dab5c21f384226df96e83884cfcecb59741bbfca29babea7e957edaa6" Nov 03 02:30:05.851053 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:30:05.851036 5437 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-1c9b4e-29894550-wm9w5_odh-ai-gateway-infra(be4f3bdc-1caa-4c1a-abb6-4bef99793882)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1c9b4e-29894550-wm9w5" podUID="be4f3bdc-1caa-4c1a-abb6-4bef99793882" Nov 03 02:30:06.662200 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:06.662151 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:30:07.631464 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:07.631430 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:30:08.011849 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:08.011754 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:30:08.229563 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:08.229516 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:30:09.522528 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:09.521987 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:30:09.995610 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:09.995572 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-ea5c75-openshift-default-75dc4675bf-qbhjq"] Nov 03 02:30:09.995853 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:09.995830 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-ea5c75-openshift-default-75dc4675bf-qbhjq" podUID="38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c" containerName="istio-proxy" containerID="cri-o://6803df855d7fe1bcfdeba838e0a7d23bc4ba21655ed60e3257b7ff94b84c06ba" gracePeriod=30 Nov 03 02:30:10.073120 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:10.073081 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:30:10.920282 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:10.920244 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:30:11.758112 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:11.758072 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:30:13.944867 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:13.944834 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:30:14.034783 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:30:14.034746 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 02:30:15.081906 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:15.081872 5437 scope.go:122] "RemoveContainer" containerID="f8fee504625aeddc3a99d090153134e2073acf7738279fc77ec829c3eb5955c6" Nov 03 02:30:15.263162 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:15.263102 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-ea5c75-openshift-default-75dc4675bf-qbhjq" Nov 03 02:30:15.341777 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:15.341747 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c-istio-podinfo\") pod \"38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c\" (UID: \"38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c\") " Nov 03 02:30:15.341907 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:15.341803 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c-workload-socket\") pod \"38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c\" (UID: \"38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c\") " Nov 03 02:30:15.341907 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:15.341844 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c-istio-data\") pod \"38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c\" (UID: \"38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c\") " Nov 03 02:30:15.341907 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:15.341894 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c-credential-socket\") pod \"38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c\" (UID: \"38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c\") " Nov 03 02:30:15.342080 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:15.341922 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c-workload-certs\") pod \"38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c\" (UID: \"38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c\") " Nov 03 02:30:15.342080 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:15.341959 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c-istiod-ca-cert\") pod \"38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c\" (UID: \"38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c\") " Nov 03 02:30:15.342080 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:15.342052 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c-istio-envoy\") pod \"38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c\" (UID: \"38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c\") " Nov 03 02:30:15.342254 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:15.342154 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c-kube-api-access-jb294\" (UniqueName: \"kubernetes.io/projected/38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c-kube-api-access-jb294\") pod \"38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c\" (UID: \"38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c\") " Nov 03 02:30:15.342254 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:15.342158 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c-istio-data" pod "38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c" (UID: "38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:30:15.342254 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:15.342153 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c-workload-socket" pod "38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c" (UID: "38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:30:15.342254 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:15.342199 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c-istio-token\" (UniqueName: \"kubernetes.io/projected/38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c-istio-token\") pod \"38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c\" (UID: \"38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c\") " Nov 03 02:30:15.342500 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:15.342238 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c-credential-socket" pod "38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c" (UID: "38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:30:15.342500 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:15.342262 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c-workload-certs" pod "38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c" (UID: "38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:30:15.342500 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:15.342460 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c-istiod-ca-cert" pod "38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c" (UID: "38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:30:15.342770 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:15.342746 5437 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c-workload-socket\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:30:15.342770 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:15.342774 5437 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c-istio-data\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:30:15.342973 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:15.342790 5437 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c-credential-socket\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:30:15.342973 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:15.342804 5437 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c-workload-certs\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:30:15.342973 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:15.342819 5437 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c-istiod-ca-cert\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:30:15.344513 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:15.344488 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c-istio-envoy" pod "38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c" (UID: "38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:30:15.344609 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:15.344536 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c-istio-token" pod "38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c" (UID: "38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:30:15.344694 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:15.344674 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c-kube-api-access-jb294" pod "38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c" (UID: "38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c"). InnerVolumeSpecName "kube-api-access-jb294". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:30:15.344787 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:15.344770 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c-istio-podinfo" pod "38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c" (UID: "38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:30:15.443912 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:15.443881 5437 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c-istio-podinfo\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:30:15.443912 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:15.443912 5437 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c-istio-envoy\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:30:15.443912 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:15.443925 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jb294\" (UniqueName: \"kubernetes.io/projected/38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c-kube-api-access-jb294\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:30:15.444162 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:15.443940 5437 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c-istio-token\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:30:15.930034 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:15.930002 5437 generic.go:356] "Generic (PLEG): container finished" podID="38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c" containerID="6803df855d7fe1bcfdeba838e0a7d23bc4ba21655ed60e3257b7ff94b84c06ba" exitCode=0 Nov 03 02:30:15.930227 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:15.930077 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-ea5c75-openshift-default-75dc4675bf-qbhjq" Nov 03 02:30:15.930227 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:15.930087 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ea5c75-openshift-default-75dc4675bf-qbhjq" event={"ID":"38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c","Type":"ContainerDied","Data":"6803df855d7fe1bcfdeba838e0a7d23bc4ba21655ed60e3257b7ff94b84c06ba"} Nov 03 02:30:15.930227 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:15.930165 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ea5c75-openshift-default-75dc4675bf-qbhjq" event={"ID":"38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c","Type":"ContainerDied","Data":"959d1d8c861933ea687b09f230ad814311988afb892a9b2b75d3bc0fc7e91bc6"} Nov 03 02:30:15.930227 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:15.930188 5437 scope.go:122] "RemoveContainer" containerID="6803df855d7fe1bcfdeba838e0a7d23bc4ba21655ed60e3257b7ff94b84c06ba" Nov 03 02:30:15.932317 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:15.932297 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41aa5f-29894550-kfrjc" event={"ID":"5308b4cd-02d4-4c3b-8b64-72063794535f","Type":"ContainerStarted","Data":"b0a8f7cbb84f0ca427d0aa183a23d141302c351b24348851a7f35b55ef80f617"} Nov 03 02:30:15.940877 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:15.940856 5437 scope.go:122] "RemoveContainer" containerID="79d094231d716860f9c35455796545cebee39a73796351fa8f0816d924774e07" Nov 03 02:30:15.953920 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:15.953898 5437 scope.go:122] "RemoveContainer" containerID="6803df855d7fe1bcfdeba838e0a7d23bc4ba21655ed60e3257b7ff94b84c06ba" Nov 03 02:30:15.954202 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:30:15.954179 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6803df855d7fe1bcfdeba838e0a7d23bc4ba21655ed60e3257b7ff94b84c06ba\": container with ID starting with 6803df855d7fe1bcfdeba838e0a7d23bc4ba21655ed60e3257b7ff94b84c06ba not found: ID does not exist" containerID="6803df855d7fe1bcfdeba838e0a7d23bc4ba21655ed60e3257b7ff94b84c06ba" Nov 03 02:30:15.954275 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:15.954209 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6803df855d7fe1bcfdeba838e0a7d23bc4ba21655ed60e3257b7ff94b84c06ba"} err="failed to get container status \"6803df855d7fe1bcfdeba838e0a7d23bc4ba21655ed60e3257b7ff94b84c06ba\": rpc error: code = NotFound desc = could not find container \"6803df855d7fe1bcfdeba838e0a7d23bc4ba21655ed60e3257b7ff94b84c06ba\": container with ID starting with 6803df855d7fe1bcfdeba838e0a7d23bc4ba21655ed60e3257b7ff94b84c06ba not found: ID does not exist" Nov 03 02:30:15.954275 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:15.954226 5437 scope.go:122] "RemoveContainer" containerID="79d094231d716860f9c35455796545cebee39a73796351fa8f0816d924774e07" Nov 03 02:30:15.954552 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:30:15.954529 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79d094231d716860f9c35455796545cebee39a73796351fa8f0816d924774e07\": container with ID starting with 79d094231d716860f9c35455796545cebee39a73796351fa8f0816d924774e07 not found: ID does not exist" containerID="79d094231d716860f9c35455796545cebee39a73796351fa8f0816d924774e07" Nov 03 02:30:15.954621 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:15.954559 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79d094231d716860f9c35455796545cebee39a73796351fa8f0816d924774e07"} err="failed to get container status \"79d094231d716860f9c35455796545cebee39a73796351fa8f0816d924774e07\": rpc error: code = NotFound desc = could not find container \"79d094231d716860f9c35455796545cebee39a73796351fa8f0816d924774e07\": container with ID starting with 79d094231d716860f9c35455796545cebee39a73796351fa8f0816d924774e07 not found: ID does not exist" Nov 03 02:30:16.081435 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:16.081401 5437 scope.go:122] "RemoveContainer" containerID="9a812bcaa90d3cda93472114ba731f907d245c93e196d457cf96e086a363a473" Nov 03 02:30:16.081614 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:16.081502 5437 scope.go:122] "RemoveContainer" containerID="42bccab2b976f7f0c85e2a1ce79cbb64b32daa57b49fd99b4e7dea7c449c960a" Nov 03 02:30:16.102314 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:16.102290 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-ea5c75-openshift-default-75dc4675bf-qbhjq"] Nov 03 02:30:16.105011 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:16.104990 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-ea5c75-openshift-default-75dc4675bf-qbhjq"] Nov 03 02:30:16.260775 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:16.260688 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-b6c56b-6f48cf5c6f-5mswj"] Nov 03 02:30:16.359917 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:16.359885 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-b6c56b-759c8fb4d7-54gk6"] Nov 03 02:30:16.666860 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:16.666828 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:30:16.880790 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:16.880755 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:30:16.941691 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:16.941655 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-434b6a-29894550-ltk6s" event={"ID":"1ab7c82f-91ef-4685-99bb-cdb46cd0f1b1","Type":"ContainerStarted","Data":"3d87dc1ff683c3d584f3fa9edd224ccce0045436116f708808b1cf9d98eb31d7"} Nov 03 02:30:16.945016 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:16.944989 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894550-tx85q" event={"ID":"98c0de5e-7908-451f-aa0d-fc9e732a4ed7","Type":"ContainerStarted","Data":"1f7e93fcff24783516e7dba7fbd875b9da8517d3b7704141ea6b94af4c6cf2fa"} Nov 03 02:30:16.946877 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:16.946856 5437 generic.go:356] "Generic (PLEG): container finished" podID="5308b4cd-02d4-4c3b-8b64-72063794535f" containerID="b0a8f7cbb84f0ca427d0aa183a23d141302c351b24348851a7f35b55ef80f617" exitCode=60 Nov 03 02:30:16.947001 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:16.946933 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41aa5f-29894550-kfrjc" event={"ID":"5308b4cd-02d4-4c3b-8b64-72063794535f","Type":"ContainerDied","Data":"b0a8f7cbb84f0ca427d0aa183a23d141302c351b24348851a7f35b55ef80f617"} Nov 03 02:30:16.947001 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:16.946994 5437 scope.go:122] "RemoveContainer" containerID="f8fee504625aeddc3a99d090153134e2073acf7738279fc77ec829c3eb5955c6" Nov 03 02:30:16.947328 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:16.947313 5437 scope.go:122] "RemoveContainer" containerID="b0a8f7cbb84f0ca427d0aa183a23d141302c351b24348851a7f35b55ef80f617" Nov 03 02:30:16.947599 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:30:16.947578 5437 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-41aa5f-29894550-kfrjc_odh-ai-gateway-infra(5308b4cd-02d4-4c3b-8b64-72063794535f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41aa5f-29894550-kfrjc" podUID="5308b4cd-02d4-4c3b-8b64-72063794535f" Nov 03 02:30:17.082414 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:17.082353 5437 scope.go:122] "RemoveContainer" containerID="e70e37d2f3554d102ed3e27c7b7e9f65b27e632ccaaa78819802d599862670a7" Nov 03 02:30:17.095340 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:17.095312 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:30:17.236254 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:17.236225 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41aa5f-29894550-kfrjc"] Nov 03 02:30:17.957521 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:17.957490 5437 generic.go:356] "Generic (PLEG): container finished" podID="1ab7c82f-91ef-4685-99bb-cdb46cd0f1b1" containerID="3d87dc1ff683c3d584f3fa9edd224ccce0045436116f708808b1cf9d98eb31d7" exitCode=60 Nov 03 02:30:17.957715 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:17.957562 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-434b6a-29894550-ltk6s" event={"ID":"1ab7c82f-91ef-4685-99bb-cdb46cd0f1b1","Type":"ContainerDied","Data":"3d87dc1ff683c3d584f3fa9edd224ccce0045436116f708808b1cf9d98eb31d7"} Nov 03 02:30:17.957715 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:17.957625 5437 scope.go:122] "RemoveContainer" containerID="9a812bcaa90d3cda93472114ba731f907d245c93e196d457cf96e086a363a473" Nov 03 02:30:17.958051 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:17.958029 5437 scope.go:122] "RemoveContainer" containerID="3d87dc1ff683c3d584f3fa9edd224ccce0045436116f708808b1cf9d98eb31d7" Nov 03 02:30:17.958584 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:30:17.958553 5437 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-434b6a-29894550-ltk6s_odh-ai-gateway-infra(1ab7c82f-91ef-4685-99bb-cdb46cd0f1b1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-434b6a-29894550-ltk6s" podUID="1ab7c82f-91ef-4685-99bb-cdb46cd0f1b1" Nov 03 02:30:17.959905 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:17.959880 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894550-fdxp4" event={"ID":"e52dc6a2-2ad8-4e48-89b2-94530d8b248f","Type":"ContainerStarted","Data":"ac38bc4edd87c045d827003cc34b3b677912266f977e8249443ea660334a4ec4"} Nov 03 02:30:17.961696 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:17.961674 5437 generic.go:356] "Generic (PLEG): container finished" podID="98c0de5e-7908-451f-aa0d-fc9e732a4ed7" containerID="1f7e93fcff24783516e7dba7fbd875b9da8517d3b7704141ea6b94af4c6cf2fa" exitCode=60 Nov 03 02:30:17.961856 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:17.961744 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894550-tx85q" event={"ID":"98c0de5e-7908-451f-aa0d-fc9e732a4ed7","Type":"ContainerDied","Data":"1f7e93fcff24783516e7dba7fbd875b9da8517d3b7704141ea6b94af4c6cf2fa"} Nov 03 02:30:17.962126 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:17.962107 5437 scope.go:122] "RemoveContainer" containerID="1f7e93fcff24783516e7dba7fbd875b9da8517d3b7704141ea6b94af4c6cf2fa" Nov 03 02:30:17.962425 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:30:17.962399 5437 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29894550-tx85q_odh-ai-gateway-infra(98c0de5e-7908-451f-aa0d-fc9e732a4ed7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894550-tx85q" podUID="98c0de5e-7908-451f-aa0d-fc9e732a4ed7" Nov 03 02:30:17.972841 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:17.972817 5437 scope.go:122] "RemoveContainer" containerID="42bccab2b976f7f0c85e2a1ce79cbb64b32daa57b49fd99b4e7dea7c449c960a" Nov 03 02:30:18.048246 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:18.046344 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:30:18.071931 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:18.071901 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-434b6a-29894550-ltk6s"] Nov 03 02:30:18.083438 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:18.082151 5437 scope.go:122] "RemoveContainer" containerID="85f1b44dab5c21f384226df96e83884cfcecb59741bbfca29babea7e957edaa6" Nov 03 02:30:18.098268 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:18.098238 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c" path="/var/lib/kubelet/pods/38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c/volumes" Nov 03 02:30:18.160748 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:18.160727 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41aa5f-29894550-kfrjc" Nov 03 02:30:18.278317 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:18.278222 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5308b4cd-02d4-4c3b-8b64-72063794535f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5308b4cd-02d4-4c3b-8b64-72063794535f-maas-api-service-ca\") pod \"5308b4cd-02d4-4c3b-8b64-72063794535f\" (UID: \"5308b4cd-02d4-4c3b-8b64-72063794535f\") " Nov 03 02:30:18.278782 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:18.278326 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5308b4cd-02d4-4c3b-8b64-72063794535f-kube-api-access-4cwt4\" (UniqueName: \"kubernetes.io/projected/5308b4cd-02d4-4c3b-8b64-72063794535f-kube-api-access-4cwt4\") pod \"5308b4cd-02d4-4c3b-8b64-72063794535f\" (UID: \"5308b4cd-02d4-4c3b-8b64-72063794535f\") " Nov 03 02:30:18.278782 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:18.278623 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5308b4cd-02d4-4c3b-8b64-72063794535f-maas-api-service-ca" pod "5308b4cd-02d4-4c3b-8b64-72063794535f" (UID: "5308b4cd-02d4-4c3b-8b64-72063794535f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:30:18.279188 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:18.279162 5437 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5308b4cd-02d4-4c3b-8b64-72063794535f-maas-api-service-ca\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:30:18.280864 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:18.280838 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5308b4cd-02d4-4c3b-8b64-72063794535f-kube-api-access-4cwt4" pod "5308b4cd-02d4-4c3b-8b64-72063794535f" (UID: "5308b4cd-02d4-4c3b-8b64-72063794535f"). InnerVolumeSpecName "kube-api-access-4cwt4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:30:18.306762 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:18.306727 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894550-tx85q"] Nov 03 02:30:18.380276 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:18.380238 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cwt4\" (UniqueName: \"kubernetes.io/projected/5308b4cd-02d4-4c3b-8b64-72063794535f-kube-api-access-4cwt4\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:30:18.471850 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:18.471814 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:30:18.971968 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:18.971933 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41aa5f-29894550-kfrjc" Nov 03 02:30:18.972185 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:18.971965 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41aa5f-29894550-kfrjc" event={"ID":"5308b4cd-02d4-4c3b-8b64-72063794535f","Type":"ContainerDied","Data":"60ed9e097ca921dbf33a458e1ef30e50a8d23a9498dc26e01374aac95c085d13"} Nov 03 02:30:18.972185 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:18.972065 5437 scope.go:122] "RemoveContainer" containerID="b0a8f7cbb84f0ca427d0aa183a23d141302c351b24348851a7f35b55ef80f617" Nov 03 02:30:18.977719 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:18.977675 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1c9b4e-29894550-wm9w5" event={"ID":"be4f3bdc-1caa-4c1a-abb6-4bef99793882","Type":"ContainerStarted","Data":"7a3559373bdf60d5f145301042e20b6170ccb334e71edca2d3882ef65e90f67b"} Nov 03 02:30:18.980705 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:18.980677 5437 generic.go:356] "Generic (PLEG): container finished" podID="e52dc6a2-2ad8-4e48-89b2-94530d8b248f" containerID="ac38bc4edd87c045d827003cc34b3b677912266f977e8249443ea660334a4ec4" exitCode=60 Nov 03 02:30:18.980840 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:18.980795 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894550-fdxp4" event={"ID":"e52dc6a2-2ad8-4e48-89b2-94530d8b248f","Type":"ContainerDied","Data":"ac38bc4edd87c045d827003cc34b3b677912266f977e8249443ea660334a4ec4"} Nov 03 02:30:18.981185 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:18.981162 5437 scope.go:122] "RemoveContainer" containerID="ac38bc4edd87c045d827003cc34b3b677912266f977e8249443ea660334a4ec4" Nov 03 02:30:18.981807 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:30:18.981636 5437 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894550-fdxp4_odh-ai-gateway-infra(e52dc6a2-2ad8-4e48-89b2-94530d8b248f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894550-fdxp4" podUID="e52dc6a2-2ad8-4e48-89b2-94530d8b248f" Nov 03 02:30:18.985553 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:18.985523 5437 scope.go:122] "RemoveContainer" containerID="e70e37d2f3554d102ed3e27c7b7e9f65b27e632ccaaa78819802d599862670a7" Nov 03 02:30:19.202406 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:19.202353 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-434b6a-29894550-ltk6s" Nov 03 02:30:19.205027 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:19.205008 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894550-tx85q" Nov 03 02:30:19.264501 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:19.264425 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41aa5f-29894550-kfrjc"] Nov 03 02:30:19.268614 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:19.268583 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41aa5f-29894550-kfrjc"] Nov 03 02:30:19.290200 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:19.290173 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894550-fdxp4"] Nov 03 02:30:19.291842 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:19.291811 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98c0de5e-7908-451f-aa0d-fc9e732a4ed7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98c0de5e-7908-451f-aa0d-fc9e732a4ed7-maas-api-service-ca\") pod \"98c0de5e-7908-451f-aa0d-fc9e732a4ed7\" (UID: \"98c0de5e-7908-451f-aa0d-fc9e732a4ed7\") " Nov 03 02:30:19.291947 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:19.291900 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ab7c82f-91ef-4685-99bb-cdb46cd0f1b1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ab7c82f-91ef-4685-99bb-cdb46cd0f1b1-maas-api-service-ca\") pod \"1ab7c82f-91ef-4685-99bb-cdb46cd0f1b1\" (UID: \"1ab7c82f-91ef-4685-99bb-cdb46cd0f1b1\") " Nov 03 02:30:19.292004 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:19.291970 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98c0de5e-7908-451f-aa0d-fc9e732a4ed7-kube-api-access-5ggf9\" (UniqueName: \"kubernetes.io/projected/98c0de5e-7908-451f-aa0d-fc9e732a4ed7-kube-api-access-5ggf9\") pod \"98c0de5e-7908-451f-aa0d-fc9e732a4ed7\" (UID: \"98c0de5e-7908-451f-aa0d-fc9e732a4ed7\") " Nov 03 02:30:19.292004 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:19.291998 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ab7c82f-91ef-4685-99bb-cdb46cd0f1b1-kube-api-access-ndw26\" (UniqueName: \"kubernetes.io/projected/1ab7c82f-91ef-4685-99bb-cdb46cd0f1b1-kube-api-access-ndw26\") pod \"1ab7c82f-91ef-4685-99bb-cdb46cd0f1b1\" (UID: \"1ab7c82f-91ef-4685-99bb-cdb46cd0f1b1\") " Nov 03 02:30:19.292210 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:19.292182 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98c0de5e-7908-451f-aa0d-fc9e732a4ed7-maas-api-service-ca" pod "98c0de5e-7908-451f-aa0d-fc9e732a4ed7" (UID: "98c0de5e-7908-451f-aa0d-fc9e732a4ed7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:30:19.292286 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:19.292236 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ab7c82f-91ef-4685-99bb-cdb46cd0f1b1-maas-api-service-ca" pod "1ab7c82f-91ef-4685-99bb-cdb46cd0f1b1" (UID: "1ab7c82f-91ef-4685-99bb-cdb46cd0f1b1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:30:19.292442 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:19.292423 5437 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98c0de5e-7908-451f-aa0d-fc9e732a4ed7-maas-api-service-ca\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:30:19.292442 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:19.292442 5437 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ab7c82f-91ef-4685-99bb-cdb46cd0f1b1-maas-api-service-ca\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:30:19.294202 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:19.294179 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98c0de5e-7908-451f-aa0d-fc9e732a4ed7-kube-api-access-5ggf9" pod "98c0de5e-7908-451f-aa0d-fc9e732a4ed7" (UID: "98c0de5e-7908-451f-aa0d-fc9e732a4ed7"). InnerVolumeSpecName "kube-api-access-5ggf9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:30:19.294287 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:19.294210 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ab7c82f-91ef-4685-99bb-cdb46cd0f1b1-kube-api-access-ndw26" pod "1ab7c82f-91ef-4685-99bb-cdb46cd0f1b1" (UID: "1ab7c82f-91ef-4685-99bb-cdb46cd0f1b1"). InnerVolumeSpecName "kube-api-access-ndw26". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:30:19.394071 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:19.393943 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ggf9\" (UniqueName: \"kubernetes.io/projected/98c0de5e-7908-451f-aa0d-fc9e732a4ed7-kube-api-access-5ggf9\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:30:19.394349 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:19.394319 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndw26\" (UniqueName: \"kubernetes.io/projected/1ab7c82f-91ef-4685-99bb-cdb46cd0f1b1-kube-api-access-ndw26\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:30:19.991170 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:19.991141 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894550-tx85q" Nov 03 02:30:19.991408 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:19.991136 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894550-tx85q" event={"ID":"98c0de5e-7908-451f-aa0d-fc9e732a4ed7","Type":"ContainerDied","Data":"692179fdc21205e8c0477731d56216bbf8b4c4ee9e251dcf1375a26eb3360dfc"} Nov 03 02:30:19.991408 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:19.991301 5437 scope.go:122] "RemoveContainer" containerID="1f7e93fcff24783516e7dba7fbd875b9da8517d3b7704141ea6b94af4c6cf2fa" Nov 03 02:30:19.994146 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:19.994114 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-434b6a-29894550-ltk6s" Nov 03 02:30:19.994278 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:19.994152 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-434b6a-29894550-ltk6s" event={"ID":"1ab7c82f-91ef-4685-99bb-cdb46cd0f1b1","Type":"ContainerDied","Data":"3f22d176989a9b8fe7df540dc92072b6c111303d2025960c810241a65a3bb85a"} Nov 03 02:30:19.996433 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:19.996411 5437 generic.go:356] "Generic (PLEG): container finished" podID="be4f3bdc-1caa-4c1a-abb6-4bef99793882" containerID="7a3559373bdf60d5f145301042e20b6170ccb334e71edca2d3882ef65e90f67b" exitCode=60 Nov 03 02:30:19.996555 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:19.996481 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1c9b4e-29894550-wm9w5" event={"ID":"be4f3bdc-1caa-4c1a-abb6-4bef99793882","Type":"ContainerDied","Data":"7a3559373bdf60d5f145301042e20b6170ccb334e71edca2d3882ef65e90f67b"} Nov 03 02:30:19.996868 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:19.996850 5437 scope.go:122] "RemoveContainer" containerID="7a3559373bdf60d5f145301042e20b6170ccb334e71edca2d3882ef65e90f67b" Nov 03 02:30:19.997437 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:30:19.997139 5437 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-1c9b4e-29894550-wm9w5_odh-ai-gateway-infra(be4f3bdc-1caa-4c1a-abb6-4bef99793882)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1c9b4e-29894550-wm9w5" podUID="be4f3bdc-1caa-4c1a-abb6-4bef99793882" Nov 03 02:30:20.005249 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:20.005226 5437 scope.go:122] "RemoveContainer" containerID="3d87dc1ff683c3d584f3fa9edd224ccce0045436116f708808b1cf9d98eb31d7" Nov 03 02:30:20.015777 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:20.015752 5437 scope.go:122] "RemoveContainer" containerID="85f1b44dab5c21f384226df96e83884cfcecb59741bbfca29babea7e957edaa6" Nov 03 02:30:20.094227 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:20.094190 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5308b4cd-02d4-4c3b-8b64-72063794535f" path="/var/lib/kubelet/pods/5308b4cd-02d4-4c3b-8b64-72063794535f/volumes" Nov 03 02:30:20.150558 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:20.150530 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894550-fdxp4" Nov 03 02:30:20.204792 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:20.204759 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e52dc6a2-2ad8-4e48-89b2-94530d8b248f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e52dc6a2-2ad8-4e48-89b2-94530d8b248f-maas-api-service-ca\") pod \"e52dc6a2-2ad8-4e48-89b2-94530d8b248f\" (UID: \"e52dc6a2-2ad8-4e48-89b2-94530d8b248f\") " Nov 03 02:30:20.204972 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:20.204859 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e52dc6a2-2ad8-4e48-89b2-94530d8b248f-kube-api-access-kpnqq\" (UniqueName: \"kubernetes.io/projected/e52dc6a2-2ad8-4e48-89b2-94530d8b248f-kube-api-access-kpnqq\") pod \"e52dc6a2-2ad8-4e48-89b2-94530d8b248f\" (UID: \"e52dc6a2-2ad8-4e48-89b2-94530d8b248f\") " Nov 03 02:30:20.205173 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:20.205151 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e52dc6a2-2ad8-4e48-89b2-94530d8b248f-maas-api-service-ca" pod "e52dc6a2-2ad8-4e48-89b2-94530d8b248f" (UID: "e52dc6a2-2ad8-4e48-89b2-94530d8b248f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:30:20.205359 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:20.205334 5437 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e52dc6a2-2ad8-4e48-89b2-94530d8b248f-maas-api-service-ca\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:30:20.207034 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:20.207005 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e52dc6a2-2ad8-4e48-89b2-94530d8b248f-kube-api-access-kpnqq" pod "e52dc6a2-2ad8-4e48-89b2-94530d8b248f" (UID: "e52dc6a2-2ad8-4e48-89b2-94530d8b248f"). InnerVolumeSpecName "kube-api-access-kpnqq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:30:20.281513 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:20.281437 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894550-tx85q"] Nov 03 02:30:20.285519 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:20.285495 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894550-tx85q"] Nov 03 02:30:20.306796 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:20.306764 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kpnqq\" (UniqueName: \"kubernetes.io/projected/e52dc6a2-2ad8-4e48-89b2-94530d8b248f-kube-api-access-kpnqq\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:30:20.513756 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:20.513725 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1c9b4e-29894550-wm9w5"] Nov 03 02:30:20.526607 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:20.526573 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-434b6a-29894550-ltk6s"] Nov 03 02:30:20.530226 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:20.530190 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-434b6a-29894550-ltk6s"] Nov 03 02:30:21.010357 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:21.010323 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894550-fdxp4" Nov 03 02:30:21.010357 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:21.010331 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894550-fdxp4" event={"ID":"e52dc6a2-2ad8-4e48-89b2-94530d8b248f","Type":"ContainerDied","Data":"9224e75ee66e1f04addc5ea1ec2be4b15d8c1fa928109a5faa5ccfa07e941733"} Nov 03 02:30:21.010959 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:21.010930 5437 scope.go:122] "RemoveContainer" containerID="ac38bc4edd87c045d827003cc34b3b677912266f977e8249443ea660334a4ec4" Nov 03 02:30:21.156645 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:21.156614 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1c9b4e-29894550-wm9w5" Nov 03 02:30:21.217096 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:21.217061 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be4f3bdc-1caa-4c1a-abb6-4bef99793882-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be4f3bdc-1caa-4c1a-abb6-4bef99793882-maas-api-service-ca\") pod \"be4f3bdc-1caa-4c1a-abb6-4bef99793882\" (UID: \"be4f3bdc-1caa-4c1a-abb6-4bef99793882\") " Nov 03 02:30:21.217260 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:21.217246 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be4f3bdc-1caa-4c1a-abb6-4bef99793882-kube-api-access-lhfg8\" (UniqueName: \"kubernetes.io/projected/be4f3bdc-1caa-4c1a-abb6-4bef99793882-kube-api-access-lhfg8\") pod \"be4f3bdc-1caa-4c1a-abb6-4bef99793882\" (UID: \"be4f3bdc-1caa-4c1a-abb6-4bef99793882\") " Nov 03 02:30:21.217465 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:21.217440 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be4f3bdc-1caa-4c1a-abb6-4bef99793882-maas-api-service-ca" pod "be4f3bdc-1caa-4c1a-abb6-4bef99793882" (UID: "be4f3bdc-1caa-4c1a-abb6-4bef99793882"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:30:21.218217 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:21.218195 5437 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be4f3bdc-1caa-4c1a-abb6-4bef99793882-maas-api-service-ca\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:30:21.219625 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:21.219602 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be4f3bdc-1caa-4c1a-abb6-4bef99793882-kube-api-access-lhfg8" pod "be4f3bdc-1caa-4c1a-abb6-4bef99793882" (UID: "be4f3bdc-1caa-4c1a-abb6-4bef99793882"). InnerVolumeSpecName "kube-api-access-lhfg8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:30:21.243646 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:21.243610 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894550-fdxp4"] Nov 03 02:30:21.247029 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:21.247003 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894550-fdxp4"] Nov 03 02:30:21.261777 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:21.261706 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-b6c56b-6f48cf5c6f-5mswj" podUID="9d61f263-6bc7-40bd-ab1f-f7db9e106c31" containerName="payload-processing" containerID="cri-o://af1bab038f08217d95a863af3a325fbde233047b792ad614252b4c416e89e2d0" gracePeriod=25 Nov 03 02:30:21.319677 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:21.319646 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhfg8\" (UniqueName: \"kubernetes.io/projected/be4f3bdc-1caa-4c1a-abb6-4bef99793882-kube-api-access-lhfg8\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:30:21.364089 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:21.360839 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b6c56b-759c8fb4d7-54gk6" podUID="f1c3ee08-9d4b-406f-a183-649ab4a88536" containerName="payload-pre-processing" containerID="cri-o://376b348fe47b3d4675d44f615188e4fd96a6f1c9dedb58f846a29a1253075703" gracePeriod=25 Nov 03 02:30:21.548277 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:21.548251 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-b6c56b-6f48cf5c6f-5mswj" Nov 03 02:30:21.622001 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:21.621977 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b6c56b-759c8fb4d7-54gk6" Nov 03 02:30:21.623236 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:21.623213 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d61f263-6bc7-40bd-ab1f-f7db9e106c31-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d61f263-6bc7-40bd-ab1f-f7db9e106c31-plugins-config-volume\") pod \"9d61f263-6bc7-40bd-ab1f-f7db9e106c31\" (UID: \"9d61f263-6bc7-40bd-ab1f-f7db9e106c31\") " Nov 03 02:30:21.623357 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:21.623301 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d61f263-6bc7-40bd-ab1f-f7db9e106c31-kube-api-access-tdj86\" (UniqueName: \"kubernetes.io/projected/9d61f263-6bc7-40bd-ab1f-f7db9e106c31-kube-api-access-tdj86\") pod \"9d61f263-6bc7-40bd-ab1f-f7db9e106c31\" (UID: \"9d61f263-6bc7-40bd-ab1f-f7db9e106c31\") " Nov 03 02:30:21.623587 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:21.623566 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d61f263-6bc7-40bd-ab1f-f7db9e106c31-plugins-config-volume" pod "9d61f263-6bc7-40bd-ab1f-f7db9e106c31" (UID: "9d61f263-6bc7-40bd-ab1f-f7db9e106c31"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:30:21.623834 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:21.623814 5437 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d61f263-6bc7-40bd-ab1f-f7db9e106c31-plugins-config-volume\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:30:21.625306 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:21.625286 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d61f263-6bc7-40bd-ab1f-f7db9e106c31-kube-api-access-tdj86" pod "9d61f263-6bc7-40bd-ab1f-f7db9e106c31" (UID: "9d61f263-6bc7-40bd-ab1f-f7db9e106c31"). InnerVolumeSpecName "kube-api-access-tdj86". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:30:21.724809 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:21.724776 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1c3ee08-9d4b-406f-a183-649ab4a88536-kube-api-access-h7fzb\" (UniqueName: \"kubernetes.io/projected/f1c3ee08-9d4b-406f-a183-649ab4a88536-kube-api-access-h7fzb\") pod \"f1c3ee08-9d4b-406f-a183-649ab4a88536\" (UID: \"f1c3ee08-9d4b-406f-a183-649ab4a88536\") " Nov 03 02:30:21.724988 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:21.724929 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1c3ee08-9d4b-406f-a183-649ab4a88536-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1c3ee08-9d4b-406f-a183-649ab4a88536-plugins-config-volume\") pod \"f1c3ee08-9d4b-406f-a183-649ab4a88536\" (UID: \"f1c3ee08-9d4b-406f-a183-649ab4a88536\") " Nov 03 02:30:21.725269 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:21.725253 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdj86\" (UniqueName: \"kubernetes.io/projected/9d61f263-6bc7-40bd-ab1f-f7db9e106c31-kube-api-access-tdj86\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:30:21.725357 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:21.725309 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1c3ee08-9d4b-406f-a183-649ab4a88536-plugins-config-volume" pod "f1c3ee08-9d4b-406f-a183-649ab4a88536" (UID: "f1c3ee08-9d4b-406f-a183-649ab4a88536"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:30:21.726829 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:21.726809 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1c3ee08-9d4b-406f-a183-649ab4a88536-kube-api-access-h7fzb" pod "f1c3ee08-9d4b-406f-a183-649ab4a88536" (UID: "f1c3ee08-9d4b-406f-a183-649ab4a88536"). InnerVolumeSpecName "kube-api-access-h7fzb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:30:21.826034 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:21.825997 5437 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1c3ee08-9d4b-406f-a183-649ab4a88536-plugins-config-volume\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:30:21.826034 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:21.826030 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7fzb\" (UniqueName: \"kubernetes.io/projected/f1c3ee08-9d4b-406f-a183-649ab4a88536-kube-api-access-h7fzb\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:30:21.985839 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:21.985804 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:30:22.020351 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:22.020316 5437 generic.go:356] "Generic (PLEG): container finished" podID="f1c3ee08-9d4b-406f-a183-649ab4a88536" containerID="376b348fe47b3d4675d44f615188e4fd96a6f1c9dedb58f846a29a1253075703" exitCode=0 Nov 03 02:30:22.020539 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:22.020391 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b6c56b-759c8fb4d7-54gk6" Nov 03 02:30:22.020539 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:22.020398 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b6c56b-759c8fb4d7-54gk6" event={"ID":"f1c3ee08-9d4b-406f-a183-649ab4a88536","Type":"ContainerDied","Data":"376b348fe47b3d4675d44f615188e4fd96a6f1c9dedb58f846a29a1253075703"} Nov 03 02:30:22.020539 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:22.020460 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b6c56b-759c8fb4d7-54gk6" event={"ID":"f1c3ee08-9d4b-406f-a183-649ab4a88536","Type":"ContainerDied","Data":"3235306881f8582a423a054cc83d0e06c8619a4ce528dae5f423e18afd7ffd7a"} Nov 03 02:30:22.020539 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:22.020491 5437 scope.go:122] "RemoveContainer" containerID="376b348fe47b3d4675d44f615188e4fd96a6f1c9dedb58f846a29a1253075703" Nov 03 02:30:22.022153 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:22.022107 5437 generic.go:356] "Generic (PLEG): container finished" podID="9d61f263-6bc7-40bd-ab1f-f7db9e106c31" containerID="af1bab038f08217d95a863af3a325fbde233047b792ad614252b4c416e89e2d0" exitCode=0 Nov 03 02:30:22.022264 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:22.022146 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-b6c56b-6f48cf5c6f-5mswj" event={"ID":"9d61f263-6bc7-40bd-ab1f-f7db9e106c31","Type":"ContainerDied","Data":"af1bab038f08217d95a863af3a325fbde233047b792ad614252b4c416e89e2d0"} Nov 03 02:30:22.022264 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:22.022202 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-b6c56b-6f48cf5c6f-5mswj" event={"ID":"9d61f263-6bc7-40bd-ab1f-f7db9e106c31","Type":"ContainerDied","Data":"53a2a507877c1af26e0c046fa3084c1ee6421d4e4115c96a1c58d56e29d2cf8c"} Nov 03 02:30:22.022264 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:22.022232 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-b6c56b-6f48cf5c6f-5mswj" Nov 03 02:30:22.025862 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:22.025843 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1c9b4e-29894550-wm9w5" Nov 03 02:30:22.025862 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:22.025854 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1c9b4e-29894550-wm9w5" event={"ID":"be4f3bdc-1caa-4c1a-abb6-4bef99793882","Type":"ContainerDied","Data":"f5ae21a4e829cf9e686bfd9859dd2664297b2ed345cb6aa16fe5374d3f5ee909"} Nov 03 02:30:22.033534 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:22.033511 5437 scope.go:122] "RemoveContainer" containerID="376b348fe47b3d4675d44f615188e4fd96a6f1c9dedb58f846a29a1253075703" Nov 03 02:30:22.033826 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:30:22.033796 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"376b348fe47b3d4675d44f615188e4fd96a6f1c9dedb58f846a29a1253075703\": container with ID starting with 376b348fe47b3d4675d44f615188e4fd96a6f1c9dedb58f846a29a1253075703 not found: ID does not exist" containerID="376b348fe47b3d4675d44f615188e4fd96a6f1c9dedb58f846a29a1253075703" Nov 03 02:30:22.033898 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:22.033833 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"376b348fe47b3d4675d44f615188e4fd96a6f1c9dedb58f846a29a1253075703"} err="failed to get container status \"376b348fe47b3d4675d44f615188e4fd96a6f1c9dedb58f846a29a1253075703\": rpc error: code = NotFound desc = could not find container \"376b348fe47b3d4675d44f615188e4fd96a6f1c9dedb58f846a29a1253075703\": container with ID starting with 376b348fe47b3d4675d44f615188e4fd96a6f1c9dedb58f846a29a1253075703 not found: ID does not exist" Nov 03 02:30:22.033898 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:22.033852 5437 scope.go:122] "RemoveContainer" containerID="af1bab038f08217d95a863af3a325fbde233047b792ad614252b4c416e89e2d0" Nov 03 02:30:22.043397 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:22.043350 5437 scope.go:122] "RemoveContainer" containerID="af1bab038f08217d95a863af3a325fbde233047b792ad614252b4c416e89e2d0" Nov 03 02:30:22.043770 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:30:22.043753 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af1bab038f08217d95a863af3a325fbde233047b792ad614252b4c416e89e2d0\": container with ID starting with af1bab038f08217d95a863af3a325fbde233047b792ad614252b4c416e89e2d0 not found: ID does not exist" containerID="af1bab038f08217d95a863af3a325fbde233047b792ad614252b4c416e89e2d0" Nov 03 02:30:22.043840 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:22.043779 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af1bab038f08217d95a863af3a325fbde233047b792ad614252b4c416e89e2d0"} err="failed to get container status \"af1bab038f08217d95a863af3a325fbde233047b792ad614252b4c416e89e2d0\": rpc error: code = NotFound desc = could not find container \"af1bab038f08217d95a863af3a325fbde233047b792ad614252b4c416e89e2d0\": container with ID starting with af1bab038f08217d95a863af3a325fbde233047b792ad614252b4c416e89e2d0 not found: ID does not exist" Nov 03 02:30:22.043840 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:22.043796 5437 scope.go:122] "RemoveContainer" containerID="7a3559373bdf60d5f145301042e20b6170ccb334e71edca2d3882ef65e90f67b" Nov 03 02:30:22.091000 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:22.090941 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ab7c82f-91ef-4685-99bb-cdb46cd0f1b1" path="/var/lib/kubelet/pods/1ab7c82f-91ef-4685-99bb-cdb46cd0f1b1/volumes" Nov 03 02:30:22.091436 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:22.091419 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98c0de5e-7908-451f-aa0d-fc9e732a4ed7" path="/var/lib/kubelet/pods/98c0de5e-7908-451f-aa0d-fc9e732a4ed7/volumes" Nov 03 02:30:22.091830 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:22.091817 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e52dc6a2-2ad8-4e48-89b2-94530d8b248f" path="/var/lib/kubelet/pods/e52dc6a2-2ad8-4e48-89b2-94530d8b248f/volumes" Nov 03 02:30:22.190314 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:22.190278 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1c9b4e-29894550-wm9w5"] Nov 03 02:30:22.194421 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:22.194388 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1c9b4e-29894550-wm9w5"] Nov 03 02:30:22.306890 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:22.306854 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-b6c56b-759c8fb4d7-54gk6"] Nov 03 02:30:22.311017 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:22.310991 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-b6c56b-759c8fb4d7-54gk6"] Nov 03 02:30:22.424170 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:22.424085 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-b6c56b-6f48cf5c6f-5mswj"] Nov 03 02:30:22.427674 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:22.427650 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-b6c56b-6f48cf5c6f-5mswj"] Nov 03 02:30:24.088912 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:24.088876 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d61f263-6bc7-40bd-ab1f-f7db9e106c31" path="/var/lib/kubelet/pods/9d61f263-6bc7-40bd-ab1f-f7db9e106c31/volumes" Nov 03 02:30:24.089460 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:24.089440 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be4f3bdc-1caa-4c1a-abb6-4bef99793882" path="/var/lib/kubelet/pods/be4f3bdc-1caa-4c1a-abb6-4bef99793882/volumes" Nov 03 02:30:24.089839 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:24.089826 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1c3ee08-9d4b-406f-a183-649ab4a88536" path="/var/lib/kubelet/pods/f1c3ee08-9d4b-406f-a183-649ab4a88536/volumes" Nov 03 02:30:24.278158 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:24.278125 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:30:27.244661 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:27.244624 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:30:31.218649 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:31.218611 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:30:31.423420 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:31.423386 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:30:31.706916 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:31.706876 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:30:36.429129 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:36.429075 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:30:41.201583 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:41.201542 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:30:42.310916 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:42.310871 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:30:44.040715 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:30:44.040678 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 02:30:46.085925 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:46.085857 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:30:48.390312 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:48.390219 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:30:50.874834 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:50.874788 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:30:54.208465 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:54.208422 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:30:55.574736 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:55.574689 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:30:59.739259 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:30:59.739221 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:31:00.482076 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:31:00.482040 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:31:01.438134 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:31:01.438096 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:31:05.441013 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:31:05.440976 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:31:06.193686 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:31:06.193617 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:31:09.911235 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:31:09.911192 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:31:10.120794 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:31:10.120752 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:31:10.778706 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:31:10.778671 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:31:11.031493 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:31:11.031392 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:31:11.309915 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:31:11.309880 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:31:12.156932 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:31:12.156898 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:31:12.690507 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:31:12.690462 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:31:14.036040 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:31:14.036003 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 02:31:16.163075 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:31:16.163014 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:31:18.392939 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:31:18.392903 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e40047-5658977b99-28lz4"] Nov 03 02:31:18.452710 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:31:18.452671 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:31:18.800660 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:31:18.800625 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:31:19.006196 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:31:19.006160 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:31:20.174770 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:31:20.174734 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:31:20.387717 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:31:20.387686 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:31:21.090844 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:31:21.090803 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:31:21.295349 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:31:21.295303 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:31:23.393677 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:31:23.393628 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-e40047-5658977b99-28lz4" podUID="3004112d-86f4-4c2e-9e49-1f8e08c99933" containerName="payload-processing" containerID="cri-o://e4089511b04f82f57fedfaa48981facf928554a2b771c5653101f5312e72f99b" gracePeriod=25 Nov 03 02:31:23.667516 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:31:23.667481 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-e40047-5658977b99-28lz4" Nov 03 02:31:23.797236 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:31:23.797204 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3004112d-86f4-4c2e-9e49-1f8e08c99933-kube-api-access-4cbb2\" (UniqueName: \"kubernetes.io/projected/3004112d-86f4-4c2e-9e49-1f8e08c99933-kube-api-access-4cbb2\") pod \"3004112d-86f4-4c2e-9e49-1f8e08c99933\" (UID: \"3004112d-86f4-4c2e-9e49-1f8e08c99933\") " Nov 03 02:31:23.797474 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:31:23.797328 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3004112d-86f4-4c2e-9e49-1f8e08c99933-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3004112d-86f4-4c2e-9e49-1f8e08c99933-plugins-config-volume\") pod \"3004112d-86f4-4c2e-9e49-1f8e08c99933\" (UID: \"3004112d-86f4-4c2e-9e49-1f8e08c99933\") " Nov 03 02:31:23.797797 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:31:23.797763 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3004112d-86f4-4c2e-9e49-1f8e08c99933-plugins-config-volume" pod "3004112d-86f4-4c2e-9e49-1f8e08c99933" (UID: "3004112d-86f4-4c2e-9e49-1f8e08c99933"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:31:23.798000 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:31:23.797972 5437 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3004112d-86f4-4c2e-9e49-1f8e08c99933-plugins-config-volume\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:31:23.799672 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:31:23.799644 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3004112d-86f4-4c2e-9e49-1f8e08c99933-kube-api-access-4cbb2" pod "3004112d-86f4-4c2e-9e49-1f8e08c99933" (UID: "3004112d-86f4-4c2e-9e49-1f8e08c99933"). InnerVolumeSpecName "kube-api-access-4cbb2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:31:23.899350 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:31:23.899271 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cbb2\" (UniqueName: \"kubernetes.io/projected/3004112d-86f4-4c2e-9e49-1f8e08c99933-kube-api-access-4cbb2\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:31:24.462588 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:31:24.462552 5437 generic.go:356] "Generic (PLEG): container finished" podID="3004112d-86f4-4c2e-9e49-1f8e08c99933" containerID="e4089511b04f82f57fedfaa48981facf928554a2b771c5653101f5312e72f99b" exitCode=0 Nov 03 02:31:24.463120 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:31:24.462602 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-e40047-5658977b99-28lz4" Nov 03 02:31:24.463120 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:31:24.462641 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-e40047-5658977b99-28lz4" event={"ID":"3004112d-86f4-4c2e-9e49-1f8e08c99933","Type":"ContainerDied","Data":"e4089511b04f82f57fedfaa48981facf928554a2b771c5653101f5312e72f99b"} Nov 03 02:31:24.463120 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:31:24.462711 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-e40047-5658977b99-28lz4" event={"ID":"3004112d-86f4-4c2e-9e49-1f8e08c99933","Type":"ContainerDied","Data":"32e73446fef5311dc1afbfad3ddc0ac93c56bff6c360cf742747cc688ae2de72"} Nov 03 02:31:24.463120 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:31:24.462746 5437 scope.go:122] "RemoveContainer" containerID="e4089511b04f82f57fedfaa48981facf928554a2b771c5653101f5312e72f99b" Nov 03 02:31:24.472816 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:31:24.472795 5437 scope.go:122] "RemoveContainer" containerID="e4089511b04f82f57fedfaa48981facf928554a2b771c5653101f5312e72f99b" Nov 03 02:31:24.473101 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:31:24.473077 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4089511b04f82f57fedfaa48981facf928554a2b771c5653101f5312e72f99b\": container with ID starting with e4089511b04f82f57fedfaa48981facf928554a2b771c5653101f5312e72f99b not found: ID does not exist" containerID="e4089511b04f82f57fedfaa48981facf928554a2b771c5653101f5312e72f99b" Nov 03 02:31:24.473147 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:31:24.473109 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4089511b04f82f57fedfaa48981facf928554a2b771c5653101f5312e72f99b"} err="failed to get container status \"e4089511b04f82f57fedfaa48981facf928554a2b771c5653101f5312e72f99b\": rpc error: code = NotFound desc = could not find container \"e4089511b04f82f57fedfaa48981facf928554a2b771c5653101f5312e72f99b\": container with ID starting with e4089511b04f82f57fedfaa48981facf928554a2b771c5653101f5312e72f99b not found: ID does not exist" Nov 03 02:31:24.513818 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:31:24.513781 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:31:24.632273 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:31:24.632234 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e40047-5658977b99-28lz4"] Nov 03 02:31:24.683646 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:31:24.683612 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e40047-5658977b99-28lz4"] Nov 03 02:31:25.501191 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:31:25.501150 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:31:26.092868 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:31:26.092821 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3004112d-86f4-4c2e-9e49-1f8e08c99933" path="/var/lib/kubelet/pods/3004112d-86f4-4c2e-9e49-1f8e08c99933/volumes" Nov 03 02:31:27.192068 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:31:27.191966 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:31:31.981143 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:31:31.981097 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:31:33.499744 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:31:33.499699 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:31:33.781304 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:31:33.781214 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:31:35.033649 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:31:35.033609 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:31:35.508490 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:31:35.508441 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:31:36.915721 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:31:36.915680 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:31:41.979469 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:31:41.979429 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:31:44.035517 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:31:44.035471 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 02:31:46.721786 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:31:46.721746 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:31:47.269977 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:31:47.269941 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:31:49.351283 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:31:49.351245 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:31:51.443932 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:31:51.443893 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:31:53.258234 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:31:53.258196 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:31:55.560845 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:31:55.560808 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:31:56.191180 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:31:56.191139 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:32:00.829388 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:00.829344 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:32:01.088207 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:01.088101 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:32:05.691121 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:05.691081 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:32:05.946757 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:05.946666 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:32:10.714693 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:10.714647 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:32:14.036267 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:32:14.036227 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 02:32:15.486817 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:15.486769 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:32:18.898060 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:18.897966 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:32:19.150507 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:19.150406 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:32:19.358788 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:19.358751 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:32:19.779036 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:19.778999 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:32:21.478422 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:21.478386 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:32:23.031456 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:23.031418 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:32:23.233270 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:23.233232 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:32:24.552359 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:24.552320 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:32:25.055965 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:25.055925 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-e40047-openshift-default-7797d7b96f-qtjtx"] Nov 03 02:32:25.056180 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:25.056154 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-e40047-openshift-default-7797d7b96f-qtjtx" podUID="953b62df-8d68-4d1e-b0b2-08b87f292998" containerName="istio-proxy" containerID="cri-o://6210fe4358a0b67625a90cf5ae7e8b8e82bedb784b282a235424b8e6ea6267eb" gracePeriod=30 Nov 03 02:32:25.099118 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:25.099085 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:32:25.589317 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:25.589278 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:32:25.800257 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:25.800218 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:32:26.019304 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:26.019223 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:32:26.546796 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:26.546648 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:32:26.750773 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:26.750732 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:32:26.959616 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:26.959507 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:32:29.894787 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:29.894752 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-c34879/rate-test-model-c34879-kserve-5c9b7d989-bbjkc"] Nov 03 02:32:29.895396 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:29.895355 5437 cpu_manager.go:414] "containerMap: removing container" podUID="be4f3bdc-1caa-4c1a-abb6-4bef99793882" containerName="cleanup" Nov 03 02:32:29.895396 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:29.895384 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be4f3bdc-1caa-4c1a-abb6-4bef99793882" containerName="cleanup" Nov 03 02:32:29.895497 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:29.895402 5437 cpu_manager.go:414] "containerMap: removing container" podUID="1ab7c82f-91ef-4685-99bb-cdb46cd0f1b1" containerName="cleanup" Nov 03 02:32:29.895497 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:29.895411 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ab7c82f-91ef-4685-99bb-cdb46cd0f1b1" containerName="cleanup" Nov 03 02:32:29.895497 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:29.895418 5437 cpu_manager.go:414] "containerMap: removing container" podUID="98c0de5e-7908-451f-aa0d-fc9e732a4ed7" containerName="cleanup" Nov 03 02:32:29.895497 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:29.895423 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98c0de5e-7908-451f-aa0d-fc9e732a4ed7" containerName="cleanup" Nov 03 02:32:29.895497 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:29.895436 5437 cpu_manager.go:414] "containerMap: removing container" podUID="5308b4cd-02d4-4c3b-8b64-72063794535f" containerName="cleanup" Nov 03 02:32:29.895497 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:29.895440 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5308b4cd-02d4-4c3b-8b64-72063794535f" containerName="cleanup" Nov 03 02:32:29.895497 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:29.895446 5437 cpu_manager.go:414] "containerMap: removing container" podUID="f1c3ee08-9d4b-406f-a183-649ab4a88536" containerName="payload-pre-processing" Nov 03 02:32:29.895497 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:29.895452 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1c3ee08-9d4b-406f-a183-649ab4a88536" containerName="payload-pre-processing" Nov 03 02:32:29.895497 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:29.895469 5437 cpu_manager.go:414] "containerMap: removing container" podUID="3004112d-86f4-4c2e-9e49-1f8e08c99933" containerName="payload-processing" Nov 03 02:32:29.895497 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:29.895473 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3004112d-86f4-4c2e-9e49-1f8e08c99933" containerName="payload-processing" Nov 03 02:32:29.895497 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:29.895490 5437 cpu_manager.go:414] "containerMap: removing container" podUID="1ab7c82f-91ef-4685-99bb-cdb46cd0f1b1" containerName="cleanup" Nov 03 02:32:29.895497 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:29.895496 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ab7c82f-91ef-4685-99bb-cdb46cd0f1b1" containerName="cleanup" Nov 03 02:32:29.895872 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:29.895505 5437 cpu_manager.go:414] "containerMap: removing container" podUID="5308b4cd-02d4-4c3b-8b64-72063794535f" containerName="cleanup" Nov 03 02:32:29.895872 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:29.895510 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5308b4cd-02d4-4c3b-8b64-72063794535f" containerName="cleanup" Nov 03 02:32:29.895872 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:29.895515 5437 cpu_manager.go:414] "containerMap: removing container" podUID="1ab7c82f-91ef-4685-99bb-cdb46cd0f1b1" containerName="cleanup" Nov 03 02:32:29.895872 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:29.895520 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ab7c82f-91ef-4685-99bb-cdb46cd0f1b1" containerName="cleanup" Nov 03 02:32:29.895872 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:29.895528 5437 cpu_manager.go:414] "containerMap: removing container" podUID="5308b4cd-02d4-4c3b-8b64-72063794535f" containerName="cleanup" Nov 03 02:32:29.895872 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:29.895532 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5308b4cd-02d4-4c3b-8b64-72063794535f" containerName="cleanup" Nov 03 02:32:29.895872 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:29.895538 5437 cpu_manager.go:414] "containerMap: removing container" podUID="e52dc6a2-2ad8-4e48-89b2-94530d8b248f" containerName="cleanup" Nov 03 02:32:29.895872 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:29.895542 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e52dc6a2-2ad8-4e48-89b2-94530d8b248f" containerName="cleanup" Nov 03 02:32:29.895872 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:29.895548 5437 cpu_manager.go:414] "containerMap: removing container" podUID="be4f3bdc-1caa-4c1a-abb6-4bef99793882" containerName="cleanup" Nov 03 02:32:29.895872 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:29.895552 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be4f3bdc-1caa-4c1a-abb6-4bef99793882" containerName="cleanup" Nov 03 02:32:29.895872 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:29.895561 5437 cpu_manager.go:414] "containerMap: removing container" podUID="38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c" containerName="istio-proxy" Nov 03 02:32:29.895872 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:29.895566 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c" containerName="istio-proxy" Nov 03 02:32:29.895872 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:29.895576 5437 cpu_manager.go:414] "containerMap: removing container" podUID="38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c" containerName="istio-proxy" Nov 03 02:32:29.895872 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:29.895580 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c" containerName="istio-proxy" Nov 03 02:32:29.895872 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:29.895587 5437 cpu_manager.go:414] "containerMap: removing container" podUID="98c0de5e-7908-451f-aa0d-fc9e732a4ed7" containerName="cleanup" Nov 03 02:32:29.895872 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:29.895591 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98c0de5e-7908-451f-aa0d-fc9e732a4ed7" containerName="cleanup" Nov 03 02:32:29.895872 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:29.895599 5437 cpu_manager.go:414] "containerMap: removing container" podUID="9d61f263-6bc7-40bd-ab1f-f7db9e106c31" containerName="payload-processing" Nov 03 02:32:29.895872 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:29.895604 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d61f263-6bc7-40bd-ab1f-f7db9e106c31" containerName="payload-processing" Nov 03 02:32:29.895872 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:29.895613 5437 cpu_manager.go:414] "containerMap: removing container" podUID="e52dc6a2-2ad8-4e48-89b2-94530d8b248f" containerName="cleanup" Nov 03 02:32:29.895872 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:29.895617 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e52dc6a2-2ad8-4e48-89b2-94530d8b248f" containerName="cleanup" Nov 03 02:32:29.895872 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:29.895709 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c" containerName="istio-proxy" Nov 03 02:32:29.895872 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:29.895716 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="98c0de5e-7908-451f-aa0d-fc9e732a4ed7" containerName="cleanup" Nov 03 02:32:29.895872 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:29.895723 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d61f263-6bc7-40bd-ab1f-f7db9e106c31" containerName="payload-processing" Nov 03 02:32:29.895872 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:29.895730 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="e52dc6a2-2ad8-4e48-89b2-94530d8b248f" containerName="cleanup" Nov 03 02:32:29.895872 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:29.895736 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="be4f3bdc-1caa-4c1a-abb6-4bef99793882" containerName="cleanup" Nov 03 02:32:29.895872 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:29.895743 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="e52dc6a2-2ad8-4e48-89b2-94530d8b248f" containerName="cleanup" Nov 03 02:32:29.895872 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:29.895750 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="be4f3bdc-1caa-4c1a-abb6-4bef99793882" containerName="cleanup" Nov 03 02:32:29.895872 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:29.895755 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ab7c82f-91ef-4685-99bb-cdb46cd0f1b1" containerName="cleanup" Nov 03 02:32:29.895872 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:29.895760 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="98c0de5e-7908-451f-aa0d-fc9e732a4ed7" containerName="cleanup" Nov 03 02:32:29.895872 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:29.895766 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="5308b4cd-02d4-4c3b-8b64-72063794535f" containerName="cleanup" Nov 03 02:32:29.895872 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:29.895771 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1c3ee08-9d4b-406f-a183-649ab4a88536" containerName="payload-pre-processing" Nov 03 02:32:29.895872 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:29.895777 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="3004112d-86f4-4c2e-9e49-1f8e08c99933" containerName="payload-processing" Nov 03 02:32:29.895872 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:29.895783 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ab7c82f-91ef-4685-99bb-cdb46cd0f1b1" containerName="cleanup" Nov 03 02:32:29.895872 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:29.895788 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="5308b4cd-02d4-4c3b-8b64-72063794535f" containerName="cleanup" Nov 03 02:32:29.899111 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:29.899095 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-c34879/rate-test-model-c34879-kserve-5c9b7d989-bbjkc" Nov 03 02:32:29.952871 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:29.952834 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-c34879/rate-test-model-c34879-kserve-5c9b7d989-bbjkc"] Nov 03 02:32:30.001773 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:30.001737 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-c34879\"/\"rate-test-model-c34879-kserve-self-signed-certs\"" Nov 03 02:32:30.001990 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:30.001822 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-c34879\"/\"kube-root-ca.crt\"" Nov 03 02:32:30.004881 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:30.004858 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-c34879\"/\"openshift-service-ca.crt\"" Nov 03 02:32:30.005004 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:30.004922 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-c34879\"/\"default-dockercfg-48fzx\"" Nov 03 02:32:30.037194 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:30.037154 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff3f6c0f-5c72-4897-ac51-f9cef66c9f95-tmp-dir\") pod \"rate-test-model-c34879-kserve-5c9b7d989-bbjkc\" (UID: \"ff3f6c0f-5c72-4897-ac51-f9cef66c9f95\") " pod="ai-tenant-e2e-shared-a-w4-c34879/rate-test-model-c34879-kserve-5c9b7d989-bbjkc" Nov 03 02:32:30.037429 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:30.037347 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff3f6c0f-5c72-4897-ac51-f9cef66c9f95-home\") pod \"rate-test-model-c34879-kserve-5c9b7d989-bbjkc\" (UID: \"ff3f6c0f-5c72-4897-ac51-f9cef66c9f95\") " pod="ai-tenant-e2e-shared-a-w4-c34879/rate-test-model-c34879-kserve-5c9b7d989-bbjkc" Nov 03 02:32:30.037508 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:30.037425 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff3f6c0f-5c72-4897-ac51-f9cef66c9f95-dshm\") pod \"rate-test-model-c34879-kserve-5c9b7d989-bbjkc\" (UID: \"ff3f6c0f-5c72-4897-ac51-f9cef66c9f95\") " pod="ai-tenant-e2e-shared-a-w4-c34879/rate-test-model-c34879-kserve-5c9b7d989-bbjkc" Nov 03 02:32:30.037508 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:30.037475 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff3f6c0f-5c72-4897-ac51-f9cef66c9f95-model-cache\") pod \"rate-test-model-c34879-kserve-5c9b7d989-bbjkc\" (UID: \"ff3f6c0f-5c72-4897-ac51-f9cef66c9f95\") " pod="ai-tenant-e2e-shared-a-w4-c34879/rate-test-model-c34879-kserve-5c9b7d989-bbjkc" Nov 03 02:32:30.037620 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:30.037598 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9x7ns\" (UniqueName: \"kubernetes.io/projected/ff3f6c0f-5c72-4897-ac51-f9cef66c9f95-kube-api-access-9x7ns\") pod \"rate-test-model-c34879-kserve-5c9b7d989-bbjkc\" (UID: \"ff3f6c0f-5c72-4897-ac51-f9cef66c9f95\") " pod="ai-tenant-e2e-shared-a-w4-c34879/rate-test-model-c34879-kserve-5c9b7d989-bbjkc" Nov 03 02:32:30.037679 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:30.037632 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff3f6c0f-5c72-4897-ac51-f9cef66c9f95-tls-certs\") pod \"rate-test-model-c34879-kserve-5c9b7d989-bbjkc\" (UID: \"ff3f6c0f-5c72-4897-ac51-f9cef66c9f95\") " pod="ai-tenant-e2e-shared-a-w4-c34879/rate-test-model-c34879-kserve-5c9b7d989-bbjkc" Nov 03 02:32:30.050400 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:30.050334 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:32:30.138512 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:30.138479 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff3f6c0f-5c72-4897-ac51-f9cef66c9f95-home\") pod \"rate-test-model-c34879-kserve-5c9b7d989-bbjkc\" (UID: \"ff3f6c0f-5c72-4897-ac51-f9cef66c9f95\") " pod="ai-tenant-e2e-shared-a-w4-c34879/rate-test-model-c34879-kserve-5c9b7d989-bbjkc" Nov 03 02:32:30.138512 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:30.138515 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff3f6c0f-5c72-4897-ac51-f9cef66c9f95-dshm\") pod \"rate-test-model-c34879-kserve-5c9b7d989-bbjkc\" (UID: \"ff3f6c0f-5c72-4897-ac51-f9cef66c9f95\") " pod="ai-tenant-e2e-shared-a-w4-c34879/rate-test-model-c34879-kserve-5c9b7d989-bbjkc" Nov 03 02:32:30.138743 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:30.138546 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff3f6c0f-5c72-4897-ac51-f9cef66c9f95-model-cache\") pod \"rate-test-model-c34879-kserve-5c9b7d989-bbjkc\" (UID: \"ff3f6c0f-5c72-4897-ac51-f9cef66c9f95\") " pod="ai-tenant-e2e-shared-a-w4-c34879/rate-test-model-c34879-kserve-5c9b7d989-bbjkc" Nov 03 02:32:30.138743 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:30.138621 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9x7ns\" (UniqueName: \"kubernetes.io/projected/ff3f6c0f-5c72-4897-ac51-f9cef66c9f95-kube-api-access-9x7ns\") pod \"rate-test-model-c34879-kserve-5c9b7d989-bbjkc\" (UID: \"ff3f6c0f-5c72-4897-ac51-f9cef66c9f95\") " pod="ai-tenant-e2e-shared-a-w4-c34879/rate-test-model-c34879-kserve-5c9b7d989-bbjkc" Nov 03 02:32:30.138743 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:30.138649 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff3f6c0f-5c72-4897-ac51-f9cef66c9f95-tls-certs\") pod \"rate-test-model-c34879-kserve-5c9b7d989-bbjkc\" (UID: \"ff3f6c0f-5c72-4897-ac51-f9cef66c9f95\") " pod="ai-tenant-e2e-shared-a-w4-c34879/rate-test-model-c34879-kserve-5c9b7d989-bbjkc" Nov 03 02:32:30.138922 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:30.138749 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff3f6c0f-5c72-4897-ac51-f9cef66c9f95-tmp-dir\") pod \"rate-test-model-c34879-kserve-5c9b7d989-bbjkc\" (UID: \"ff3f6c0f-5c72-4897-ac51-f9cef66c9f95\") " pod="ai-tenant-e2e-shared-a-w4-c34879/rate-test-model-c34879-kserve-5c9b7d989-bbjkc" Nov 03 02:32:30.139032 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:30.139008 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff3f6c0f-5c72-4897-ac51-f9cef66c9f95-home\") pod \"rate-test-model-c34879-kserve-5c9b7d989-bbjkc\" (UID: \"ff3f6c0f-5c72-4897-ac51-f9cef66c9f95\") " pod="ai-tenant-e2e-shared-a-w4-c34879/rate-test-model-c34879-kserve-5c9b7d989-bbjkc" Nov 03 02:32:30.139094 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:30.139034 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff3f6c0f-5c72-4897-ac51-f9cef66c9f95-model-cache\") pod \"rate-test-model-c34879-kserve-5c9b7d989-bbjkc\" (UID: \"ff3f6c0f-5c72-4897-ac51-f9cef66c9f95\") " pod="ai-tenant-e2e-shared-a-w4-c34879/rate-test-model-c34879-kserve-5c9b7d989-bbjkc" Nov 03 02:32:30.139094 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:30.139075 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff3f6c0f-5c72-4897-ac51-f9cef66c9f95-tmp-dir\") pod \"rate-test-model-c34879-kserve-5c9b7d989-bbjkc\" (UID: \"ff3f6c0f-5c72-4897-ac51-f9cef66c9f95\") " pod="ai-tenant-e2e-shared-a-w4-c34879/rate-test-model-c34879-kserve-5c9b7d989-bbjkc" Nov 03 02:32:30.140774 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:30.140753 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff3f6c0f-5c72-4897-ac51-f9cef66c9f95-dshm\") pod \"rate-test-model-c34879-kserve-5c9b7d989-bbjkc\" (UID: \"ff3f6c0f-5c72-4897-ac51-f9cef66c9f95\") " pod="ai-tenant-e2e-shared-a-w4-c34879/rate-test-model-c34879-kserve-5c9b7d989-bbjkc" Nov 03 02:32:30.141213 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:30.141190 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff3f6c0f-5c72-4897-ac51-f9cef66c9f95-tls-certs\") pod \"rate-test-model-c34879-kserve-5c9b7d989-bbjkc\" (UID: \"ff3f6c0f-5c72-4897-ac51-f9cef66c9f95\") " pod="ai-tenant-e2e-shared-a-w4-c34879/rate-test-model-c34879-kserve-5c9b7d989-bbjkc" Nov 03 02:32:30.203321 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:30.203284 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9x7ns\" (UniqueName: \"kubernetes.io/projected/ff3f6c0f-5c72-4897-ac51-f9cef66c9f95-kube-api-access-9x7ns\") pod \"rate-test-model-c34879-kserve-5c9b7d989-bbjkc\" (UID: \"ff3f6c0f-5c72-4897-ac51-f9cef66c9f95\") " pod="ai-tenant-e2e-shared-a-w4-c34879/rate-test-model-c34879-kserve-5c9b7d989-bbjkc" Nov 03 02:32:30.212975 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:30.212945 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-c34879/rate-test-model-c34879-kserve-5c9b7d989-bbjkc" Nov 03 02:32:30.329677 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:30.329648 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e40047-openshift-default-7797d7b96f-qtjtx" Nov 03 02:32:30.445264 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:30.445173 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/953b62df-8d68-4d1e-b0b2-08b87f292998-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/953b62df-8d68-4d1e-b0b2-08b87f292998-credential-socket\") pod \"953b62df-8d68-4d1e-b0b2-08b87f292998\" (UID: \"953b62df-8d68-4d1e-b0b2-08b87f292998\") " Nov 03 02:32:30.445264 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:30.445216 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/953b62df-8d68-4d1e-b0b2-08b87f292998-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/953b62df-8d68-4d1e-b0b2-08b87f292998-istio-podinfo\") pod \"953b62df-8d68-4d1e-b0b2-08b87f292998\" (UID: \"953b62df-8d68-4d1e-b0b2-08b87f292998\") " Nov 03 02:32:30.445264 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:30.445248 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/953b62df-8d68-4d1e-b0b2-08b87f292998-istio-token\" (UniqueName: \"kubernetes.io/projected/953b62df-8d68-4d1e-b0b2-08b87f292998-istio-token\") pod \"953b62df-8d68-4d1e-b0b2-08b87f292998\" (UID: \"953b62df-8d68-4d1e-b0b2-08b87f292998\") " Nov 03 02:32:30.445584 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:30.445293 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/953b62df-8d68-4d1e-b0b2-08b87f292998-kube-api-access-px846\" (UniqueName: \"kubernetes.io/projected/953b62df-8d68-4d1e-b0b2-08b87f292998-kube-api-access-px846\") pod \"953b62df-8d68-4d1e-b0b2-08b87f292998\" (UID: \"953b62df-8d68-4d1e-b0b2-08b87f292998\") " Nov 03 02:32:30.445584 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:30.445344 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/953b62df-8d68-4d1e-b0b2-08b87f292998-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/953b62df-8d68-4d1e-b0b2-08b87f292998-workload-certs\") pod \"953b62df-8d68-4d1e-b0b2-08b87f292998\" (UID: \"953b62df-8d68-4d1e-b0b2-08b87f292998\") " Nov 03 02:32:30.445584 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:30.445464 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/953b62df-8d68-4d1e-b0b2-08b87f292998-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/953b62df-8d68-4d1e-b0b2-08b87f292998-istio-envoy\") pod \"953b62df-8d68-4d1e-b0b2-08b87f292998\" (UID: \"953b62df-8d68-4d1e-b0b2-08b87f292998\") " Nov 03 02:32:30.445584 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:30.445510 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/953b62df-8d68-4d1e-b0b2-08b87f292998-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/953b62df-8d68-4d1e-b0b2-08b87f292998-workload-socket\") pod \"953b62df-8d68-4d1e-b0b2-08b87f292998\" (UID: \"953b62df-8d68-4d1e-b0b2-08b87f292998\") " Nov 03 02:32:30.445584 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:30.445510 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/953b62df-8d68-4d1e-b0b2-08b87f292998-credential-socket" pod "953b62df-8d68-4d1e-b0b2-08b87f292998" (UID: "953b62df-8d68-4d1e-b0b2-08b87f292998"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:32:30.445584 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:30.445555 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/953b62df-8d68-4d1e-b0b2-08b87f292998-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/953b62df-8d68-4d1e-b0b2-08b87f292998-istiod-ca-cert\") pod \"953b62df-8d68-4d1e-b0b2-08b87f292998\" (UID: \"953b62df-8d68-4d1e-b0b2-08b87f292998\") " Nov 03 02:32:30.445928 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:30.445617 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/953b62df-8d68-4d1e-b0b2-08b87f292998-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/953b62df-8d68-4d1e-b0b2-08b87f292998-istio-data\") pod \"953b62df-8d68-4d1e-b0b2-08b87f292998\" (UID: \"953b62df-8d68-4d1e-b0b2-08b87f292998\") " Nov 03 02:32:30.445928 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:30.445689 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/953b62df-8d68-4d1e-b0b2-08b87f292998-workload-certs" pod "953b62df-8d68-4d1e-b0b2-08b87f292998" (UID: "953b62df-8d68-4d1e-b0b2-08b87f292998"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:32:30.445928 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:30.445860 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/953b62df-8d68-4d1e-b0b2-08b87f292998-workload-socket" pod "953b62df-8d68-4d1e-b0b2-08b87f292998" (UID: "953b62df-8d68-4d1e-b0b2-08b87f292998"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:32:30.446108 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:30.446052 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/953b62df-8d68-4d1e-b0b2-08b87f292998-istio-data" pod "953b62df-8d68-4d1e-b0b2-08b87f292998" (UID: "953b62df-8d68-4d1e-b0b2-08b87f292998"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:32:30.446229 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:30.446206 5437 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/953b62df-8d68-4d1e-b0b2-08b87f292998-workload-socket\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:32:30.446298 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:30.446234 5437 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/953b62df-8d68-4d1e-b0b2-08b87f292998-istio-data\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:32:30.446298 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:30.446250 5437 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/953b62df-8d68-4d1e-b0b2-08b87f292998-credential-socket\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:32:30.446298 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:30.446265 5437 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/953b62df-8d68-4d1e-b0b2-08b87f292998-workload-certs\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:32:30.446611 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:30.446350 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/953b62df-8d68-4d1e-b0b2-08b87f292998-istiod-ca-cert" pod "953b62df-8d68-4d1e-b0b2-08b87f292998" (UID: "953b62df-8d68-4d1e-b0b2-08b87f292998"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:32:30.447776 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:30.447752 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/953b62df-8d68-4d1e-b0b2-08b87f292998-istio-envoy" pod "953b62df-8d68-4d1e-b0b2-08b87f292998" (UID: "953b62df-8d68-4d1e-b0b2-08b87f292998"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:32:30.447894 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:30.447851 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/953b62df-8d68-4d1e-b0b2-08b87f292998-istio-token" pod "953b62df-8d68-4d1e-b0b2-08b87f292998" (UID: "953b62df-8d68-4d1e-b0b2-08b87f292998"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:32:30.447894 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:30.447884 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/953b62df-8d68-4d1e-b0b2-08b87f292998-kube-api-access-px846" pod "953b62df-8d68-4d1e-b0b2-08b87f292998" (UID: "953b62df-8d68-4d1e-b0b2-08b87f292998"). InnerVolumeSpecName "kube-api-access-px846". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:32:30.447997 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:30.447918 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/953b62df-8d68-4d1e-b0b2-08b87f292998-istio-podinfo" pod "953b62df-8d68-4d1e-b0b2-08b87f292998" (UID: "953b62df-8d68-4d1e-b0b2-08b87f292998"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:32:30.547088 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:30.547053 5437 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/953b62df-8d68-4d1e-b0b2-08b87f292998-istio-envoy\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:32:30.547088 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:30.547081 5437 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/953b62df-8d68-4d1e-b0b2-08b87f292998-istiod-ca-cert\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:32:30.547088 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:30.547093 5437 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/953b62df-8d68-4d1e-b0b2-08b87f292998-istio-podinfo\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:32:30.547325 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:30.547101 5437 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/953b62df-8d68-4d1e-b0b2-08b87f292998-istio-token\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:32:30.547325 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:30.547110 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-px846\" (UniqueName: \"kubernetes.io/projected/953b62df-8d68-4d1e-b0b2-08b87f292998-kube-api-access-px846\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:32:30.608339 ip-10-0-136-16 kubenswrapper[5437]: W1103 02:32:30.608306 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podff3f6c0f_5c72_4897_ac51_f9cef66c9f95.slice/crio-2018fbce9efbfdeec983d29e69f898d1ce3072ef085ddeb07b7f2fff5c25d587 WatchSource:0}: Error finding container 2018fbce9efbfdeec983d29e69f898d1ce3072ef085ddeb07b7f2fff5c25d587: Status 404 returned error can't find the container with id 2018fbce9efbfdeec983d29e69f898d1ce3072ef085ddeb07b7f2fff5c25d587 Nov 03 02:32:30.611777 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:30.611754 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-c34879/rate-test-model-c34879-kserve-5c9b7d989-bbjkc"] Nov 03 02:32:30.916782 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:30.916749 5437 generic.go:356] "Generic (PLEG): container finished" podID="953b62df-8d68-4d1e-b0b2-08b87f292998" containerID="6210fe4358a0b67625a90cf5ae7e8b8e82bedb784b282a235424b8e6ea6267eb" exitCode=0 Nov 03 02:32:30.917267 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:30.916803 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e40047-openshift-default-7797d7b96f-qtjtx" Nov 03 02:32:30.917267 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:30.916836 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e40047-openshift-default-7797d7b96f-qtjtx" event={"ID":"953b62df-8d68-4d1e-b0b2-08b87f292998","Type":"ContainerDied","Data":"6210fe4358a0b67625a90cf5ae7e8b8e82bedb784b282a235424b8e6ea6267eb"} Nov 03 02:32:30.917267 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:30.916894 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e40047-openshift-default-7797d7b96f-qtjtx" event={"ID":"953b62df-8d68-4d1e-b0b2-08b87f292998","Type":"ContainerDied","Data":"b3661056f0f419985185d17ab007e2d5038a38672fecfe7279f0f7b3be7f2f8b"} Nov 03 02:32:30.917267 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:30.916920 5437 scope.go:122] "RemoveContainer" containerID="6210fe4358a0b67625a90cf5ae7e8b8e82bedb784b282a235424b8e6ea6267eb" Nov 03 02:32:30.918734 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:30.918709 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-c34879/rate-test-model-c34879-kserve-5c9b7d989-bbjkc" event={"ID":"ff3f6c0f-5c72-4897-ac51-f9cef66c9f95","Type":"ContainerStarted","Data":"f234defaf896033a249ce38922ebba5775738c06eaa362a66895ecf5a88bdbfd"} Nov 03 02:32:30.918853 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:30.918746 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-c34879/rate-test-model-c34879-kserve-5c9b7d989-bbjkc" event={"ID":"ff3f6c0f-5c72-4897-ac51-f9cef66c9f95","Type":"ContainerStarted","Data":"2018fbce9efbfdeec983d29e69f898d1ce3072ef085ddeb07b7f2fff5c25d587"} Nov 03 02:32:30.928523 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:30.928504 5437 scope.go:122] "RemoveContainer" containerID="1755c83f38d36a8a35c469e652a106f39070819328e2d9222883fd42014bf067" Nov 03 02:32:30.938274 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:30.938256 5437 scope.go:122] "RemoveContainer" containerID="6210fe4358a0b67625a90cf5ae7e8b8e82bedb784b282a235424b8e6ea6267eb" Nov 03 02:32:30.938636 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:32:30.938614 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6210fe4358a0b67625a90cf5ae7e8b8e82bedb784b282a235424b8e6ea6267eb\": container with ID starting with 6210fe4358a0b67625a90cf5ae7e8b8e82bedb784b282a235424b8e6ea6267eb not found: ID does not exist" containerID="6210fe4358a0b67625a90cf5ae7e8b8e82bedb784b282a235424b8e6ea6267eb" Nov 03 02:32:30.938700 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:30.938644 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6210fe4358a0b67625a90cf5ae7e8b8e82bedb784b282a235424b8e6ea6267eb"} err="failed to get container status \"6210fe4358a0b67625a90cf5ae7e8b8e82bedb784b282a235424b8e6ea6267eb\": rpc error: code = NotFound desc = could not find container \"6210fe4358a0b67625a90cf5ae7e8b8e82bedb784b282a235424b8e6ea6267eb\": container with ID starting with 6210fe4358a0b67625a90cf5ae7e8b8e82bedb784b282a235424b8e6ea6267eb not found: ID does not exist" Nov 03 02:32:30.938700 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:30.938670 5437 scope.go:122] "RemoveContainer" containerID="1755c83f38d36a8a35c469e652a106f39070819328e2d9222883fd42014bf067" Nov 03 02:32:30.938944 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:32:30.938921 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1755c83f38d36a8a35c469e652a106f39070819328e2d9222883fd42014bf067\": container with ID starting with 1755c83f38d36a8a35c469e652a106f39070819328e2d9222883fd42014bf067 not found: ID does not exist" containerID="1755c83f38d36a8a35c469e652a106f39070819328e2d9222883fd42014bf067" Nov 03 02:32:30.939010 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:30.938954 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1755c83f38d36a8a35c469e652a106f39070819328e2d9222883fd42014bf067"} err="failed to get container status \"1755c83f38d36a8a35c469e652a106f39070819328e2d9222883fd42014bf067\": rpc error: code = NotFound desc = could not find container \"1755c83f38d36a8a35c469e652a106f39070819328e2d9222883fd42014bf067\": container with ID starting with 1755c83f38d36a8a35c469e652a106f39070819328e2d9222883fd42014bf067 not found: ID does not exist" Nov 03 02:32:31.047109 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:31.047078 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:32:31.092237 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:31.092048 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-c34879/rate-test-model-c34879-kserve-5c9b7d989-bbjkc" podStartSLOduration=2.092031172 podStartE2EDuration="2.092031172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:32:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:32:31.090570983 +0000 UTC m=+4433.120053278" watchObservedRunningTime="2026-11-03 02:32:31.092031172 +0000 UTC m=+4433.121513470" Nov 03 02:32:31.211677 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:31.211601 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-e40047-openshift-default-7797d7b96f-qtjtx"] Nov 03 02:32:31.213358 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:31.213331 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-c34879/rate-test-model-c34879-kserve-5c9b7d989-bbjkc" Nov 03 02:32:31.218977 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:31.218953 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-e40047-openshift-default-7797d7b96f-qtjtx"] Nov 03 02:32:31.254451 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:31.254421 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:32:31.510253 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:31.510155 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:32:31.772800 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:31.772706 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:32:32.090078 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:32.090043 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="953b62df-8d68-4d1e-b0b2-08b87f292998" path="/var/lib/kubelet/pods/953b62df-8d68-4d1e-b0b2-08b87f292998/volumes" Nov 03 02:32:36.777665 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:36.777624 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:32:37.953352 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:37.953301 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:32:38.159420 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:38.159381 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:32:40.213209 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:40.213169 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-c34879/rate-test-model-c34879-kserve-5c9b7d989-bbjkc" Nov 03 02:32:40.225898 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:40.225865 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-c34879/rate-test-model-c34879-kserve-5c9b7d989-bbjkc" Nov 03 02:32:41.001968 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:41.001935 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-c34879/rate-test-model-c34879-kserve-5c9b7d989-bbjkc" Nov 03 02:32:41.576779 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:41.576743 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:32:41.916867 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:41.916778 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:32:42.119508 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:42.119462 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:32:44.035433 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:32:44.035391 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 02:32:46.379859 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:46.379818 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:32:51.169814 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:51.169761 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:32:55.886328 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:32:55.886285 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:33:00.607774 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:00.607730 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:33:01.160789 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:01.160750 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:33:05.347338 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:05.347276 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:33:06.150841 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:06.150801 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:33:10.046581 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:10.046540 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:33:14.036519 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:33:14.036479 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 02:33:14.781449 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:14.781411 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:33:17.721579 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:17.721535 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:33:18.003662 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:18.003578 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:33:18.302883 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:18.302840 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:33:19.575459 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:19.575418 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:33:20.034070 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:20.033978 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:33:20.667949 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:20.667896 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:33:21.084805 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:21.084768 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:33:24.371635 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:24.371596 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:33:27.109176 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:27.109134 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:33:28.653607 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:28.653569 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:33:28.856424 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:28.856355 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:33:29.172581 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:29.172542 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-eec2ff70-55d4b96df5-zzjsd"] Nov 03 02:33:29.173225 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:29.173206 5437 cpu_manager.go:414] "containerMap: removing container" podUID="953b62df-8d68-4d1e-b0b2-08b87f292998" containerName="istio-proxy" Nov 03 02:33:29.173225 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:29.173226 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="953b62df-8d68-4d1e-b0b2-08b87f292998" containerName="istio-proxy" Nov 03 02:33:29.173358 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:29.173244 5437 cpu_manager.go:414] "containerMap: removing container" podUID="be4f3bdc-1caa-4c1a-abb6-4bef99793882" containerName="cleanup" Nov 03 02:33:29.173358 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:29.173249 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be4f3bdc-1caa-4c1a-abb6-4bef99793882" containerName="cleanup" Nov 03 02:33:29.173358 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:29.173278 5437 cpu_manager.go:414] "containerMap: removing container" podUID="e52dc6a2-2ad8-4e48-89b2-94530d8b248f" containerName="cleanup" Nov 03 02:33:29.173358 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:29.173285 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e52dc6a2-2ad8-4e48-89b2-94530d8b248f" containerName="cleanup" Nov 03 02:33:29.173358 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:29.173303 5437 cpu_manager.go:414] "containerMap: removing container" podUID="98c0de5e-7908-451f-aa0d-fc9e732a4ed7" containerName="cleanup" Nov 03 02:33:29.173358 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:29.173308 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98c0de5e-7908-451f-aa0d-fc9e732a4ed7" containerName="cleanup" Nov 03 02:33:29.173604 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:29.173424 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="38fbd6ae-90e3-4eac-aef8-8fa3ea9cb98c" containerName="istio-proxy" Nov 03 02:33:29.173604 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:29.173438 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="e52dc6a2-2ad8-4e48-89b2-94530d8b248f" containerName="cleanup" Nov 03 02:33:29.173604 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:29.173445 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="be4f3bdc-1caa-4c1a-abb6-4bef99793882" containerName="cleanup" Nov 03 02:33:29.173604 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:29.173452 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ab7c82f-91ef-4685-99bb-cdb46cd0f1b1" containerName="cleanup" Nov 03 02:33:29.173604 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:29.173458 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="98c0de5e-7908-451f-aa0d-fc9e732a4ed7" containerName="cleanup" Nov 03 02:33:29.173604 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:29.173467 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="953b62df-8d68-4d1e-b0b2-08b87f292998" containerName="istio-proxy" Nov 03 02:33:29.173604 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:29.173475 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="5308b4cd-02d4-4c3b-8b64-72063794535f" containerName="cleanup" Nov 03 02:33:29.173604 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:29.173481 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="953b62df-8d68-4d1e-b0b2-08b87f292998" containerName="istio-proxy" Nov 03 02:33:29.176955 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:29.176930 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-eec2ff70-55d4b96df5-zzjsd" Nov 03 02:33:29.230639 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:29.230599 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-eec2ff70-55d4b96df5-zzjsd"] Nov 03 02:33:29.278747 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:29.278715 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-eec2ff70\"" Nov 03 02:33:29.278920 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:29.278815 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-eec2ff70-dockercfg-9r26r\"" Nov 03 02:33:29.343524 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:29.343487 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/295d5843-32eb-47f1-b942-a71a98cb2ca7-plugins-config-volume\") pod \"payload-processing-e2e-mt-eec2ff70-55d4b96df5-zzjsd\" (UID: \"295d5843-32eb-47f1-b942-a71a98cb2ca7\") " pod="openshift-ingress/payload-processing-e2e-mt-eec2ff70-55d4b96df5-zzjsd" Nov 03 02:33:29.343722 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:29.343698 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dczhr\" (UniqueName: \"kubernetes.io/projected/295d5843-32eb-47f1-b942-a71a98cb2ca7-kube-api-access-dczhr\") pod \"payload-processing-e2e-mt-eec2ff70-55d4b96df5-zzjsd\" (UID: \"295d5843-32eb-47f1-b942-a71a98cb2ca7\") " pod="openshift-ingress/payload-processing-e2e-mt-eec2ff70-55d4b96df5-zzjsd" Nov 03 02:33:29.445268 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:29.445173 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/295d5843-32eb-47f1-b942-a71a98cb2ca7-plugins-config-volume\") pod \"payload-processing-e2e-mt-eec2ff70-55d4b96df5-zzjsd\" (UID: \"295d5843-32eb-47f1-b942-a71a98cb2ca7\") " pod="openshift-ingress/payload-processing-e2e-mt-eec2ff70-55d4b96df5-zzjsd" Nov 03 02:33:29.445482 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:29.445296 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dczhr\" (UniqueName: \"kubernetes.io/projected/295d5843-32eb-47f1-b942-a71a98cb2ca7-kube-api-access-dczhr\") pod \"payload-processing-e2e-mt-eec2ff70-55d4b96df5-zzjsd\" (UID: \"295d5843-32eb-47f1-b942-a71a98cb2ca7\") " pod="openshift-ingress/payload-processing-e2e-mt-eec2ff70-55d4b96df5-zzjsd" Nov 03 02:33:29.446016 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:29.445993 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/295d5843-32eb-47f1-b942-a71a98cb2ca7-plugins-config-volume\") pod \"payload-processing-e2e-mt-eec2ff70-55d4b96df5-zzjsd\" (UID: \"295d5843-32eb-47f1-b942-a71a98cb2ca7\") " pod="openshift-ingress/payload-processing-e2e-mt-eec2ff70-55d4b96df5-zzjsd" Nov 03 02:33:29.472548 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:29.472508 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:33:29.508059 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:29.508026 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dczhr\" (UniqueName: \"kubernetes.io/projected/295d5843-32eb-47f1-b942-a71a98cb2ca7-kube-api-access-dczhr\") pod \"payload-processing-e2e-mt-eec2ff70-55d4b96df5-zzjsd\" (UID: \"295d5843-32eb-47f1-b942-a71a98cb2ca7\") " pod="openshift-ingress/payload-processing-e2e-mt-eec2ff70-55d4b96df5-zzjsd" Nov 03 02:33:29.794093 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:29.794008 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-eec2ff70-55d4b96df5-zzjsd" Nov 03 02:33:30.171797 ip-10-0-136-16 kubenswrapper[5437]: W1103 02:33:30.171765 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod295d5843_32eb_47f1_b942_a71a98cb2ca7.slice/crio-54bc17b18480f71b11ea278e85372c9fc7a2dccd558990175e01046191600b67 WatchSource:0}: Error finding container 54bc17b18480f71b11ea278e85372c9fc7a2dccd558990175e01046191600b67: Status 404 returned error can't find the container with id 54bc17b18480f71b11ea278e85372c9fc7a2dccd558990175e01046191600b67 Nov 03 02:33:30.174419 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:30.174392 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-eec2ff70-55d4b96df5-zzjsd"] Nov 03 02:33:30.296545 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:30.296492 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-eec2ff70-55d4b96df5-zzjsd" event={"ID":"295d5843-32eb-47f1-b942-a71a98cb2ca7","Type":"ContainerStarted","Data":"5e9aab7e8457cf6bd870e5da344657638c47c1d965abbc8ce0778b70c89bbf09"} Nov 03 02:33:30.296718 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:30.296597 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-eec2ff70-55d4b96df5-zzjsd" event={"ID":"295d5843-32eb-47f1-b942-a71a98cb2ca7","Type":"ContainerStarted","Data":"54bc17b18480f71b11ea278e85372c9fc7a2dccd558990175e01046191600b67"} Nov 03 02:33:30.296770 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:30.296730 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-eec2ff70-55d4b96df5-zzjsd" Nov 03 02:33:30.417197 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:30.417148 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-eec2ff70-55d4b96df5-zzjsd" podStartSLOduration=1.417133384 podStartE2EDuration="1.417133384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:33:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:33:30.41553634 +0000 UTC m=+4492.445018639" watchObservedRunningTime="2026-11-03 02:33:30.417133384 +0000 UTC m=+4492.446615678" Nov 03 02:33:31.521628 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:31.521586 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:33:32.382277 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:32.382237 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:33:32.584178 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:32.584141 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:33:34.469737 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:34.469695 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:33:35.423262 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:35.423230 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-98a394/rate-test-model-98a394-kserve-6d7884c4c7-gp5g4"] Nov 03 02:33:35.423946 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:35.423928 5437 cpu_manager.go:414] "containerMap: removing container" podUID="953b62df-8d68-4d1e-b0b2-08b87f292998" containerName="istio-proxy" Nov 03 02:33:35.424010 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:35.423948 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="953b62df-8d68-4d1e-b0b2-08b87f292998" containerName="istio-proxy" Nov 03 02:33:35.427647 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:35.427626 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-98a394/rate-test-model-98a394-kserve-6d7884c4c7-gp5g4" Nov 03 02:33:35.482746 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:35.482706 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-98a394/rate-test-model-98a394-kserve-6d7884c4c7-gp5g4"] Nov 03 02:33:35.514351 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:35.514320 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/786969b9-72c9-4b56-91e4-97d255bbe6e4-tls-certs\") pod \"rate-test-model-98a394-kserve-6d7884c4c7-gp5g4\" (UID: \"786969b9-72c9-4b56-91e4-97d255bbe6e4\") " pod="ai-tenant-e2e-shared-a-w2-98a394/rate-test-model-98a394-kserve-6d7884c4c7-gp5g4" Nov 03 02:33:35.514351 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:35.514373 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/786969b9-72c9-4b56-91e4-97d255bbe6e4-tmp-dir\") pod \"rate-test-model-98a394-kserve-6d7884c4c7-gp5g4\" (UID: \"786969b9-72c9-4b56-91e4-97d255bbe6e4\") " pod="ai-tenant-e2e-shared-a-w2-98a394/rate-test-model-98a394-kserve-6d7884c4c7-gp5g4" Nov 03 02:33:35.514631 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:35.514456 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/786969b9-72c9-4b56-91e4-97d255bbe6e4-dshm\") pod \"rate-test-model-98a394-kserve-6d7884c4c7-gp5g4\" (UID: \"786969b9-72c9-4b56-91e4-97d255bbe6e4\") " pod="ai-tenant-e2e-shared-a-w2-98a394/rate-test-model-98a394-kserve-6d7884c4c7-gp5g4" Nov 03 02:33:35.514631 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:35.514552 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/786969b9-72c9-4b56-91e4-97d255bbe6e4-model-cache\") pod \"rate-test-model-98a394-kserve-6d7884c4c7-gp5g4\" (UID: \"786969b9-72c9-4b56-91e4-97d255bbe6e4\") " pod="ai-tenant-e2e-shared-a-w2-98a394/rate-test-model-98a394-kserve-6d7884c4c7-gp5g4" Nov 03 02:33:35.514631 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:35.514605 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8gpr\" (UniqueName: \"kubernetes.io/projected/786969b9-72c9-4b56-91e4-97d255bbe6e4-kube-api-access-v8gpr\") pod \"rate-test-model-98a394-kserve-6d7884c4c7-gp5g4\" (UID: \"786969b9-72c9-4b56-91e4-97d255bbe6e4\") " pod="ai-tenant-e2e-shared-a-w2-98a394/rate-test-model-98a394-kserve-6d7884c4c7-gp5g4" Nov 03 02:33:35.514742 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:35.514733 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/786969b9-72c9-4b56-91e4-97d255bbe6e4-home\") pod \"rate-test-model-98a394-kserve-6d7884c4c7-gp5g4\" (UID: \"786969b9-72c9-4b56-91e4-97d255bbe6e4\") " pod="ai-tenant-e2e-shared-a-w2-98a394/rate-test-model-98a394-kserve-6d7884c4c7-gp5g4" Nov 03 02:33:35.530276 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:35.530246 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-98a394\"/\"openshift-service-ca.crt\"" Nov 03 02:33:35.533263 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:35.533242 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-98a394\"/\"rate-test-model-98a394-kserve-self-signed-certs\"" Nov 03 02:33:35.533263 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:35.533255 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-98a394\"/\"kube-root-ca.crt\"" Nov 03 02:33:35.533437 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:35.533245 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-98a394\"/\"default-dockercfg-ztgfx\"" Nov 03 02:33:35.581948 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:35.581914 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:33:35.615770 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:35.615733 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/786969b9-72c9-4b56-91e4-97d255bbe6e4-home\") pod \"rate-test-model-98a394-kserve-6d7884c4c7-gp5g4\" (UID: \"786969b9-72c9-4b56-91e4-97d255bbe6e4\") " pod="ai-tenant-e2e-shared-a-w2-98a394/rate-test-model-98a394-kserve-6d7884c4c7-gp5g4" Nov 03 02:33:35.615957 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:35.615821 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/786969b9-72c9-4b56-91e4-97d255bbe6e4-tls-certs\") pod \"rate-test-model-98a394-kserve-6d7884c4c7-gp5g4\" (UID: \"786969b9-72c9-4b56-91e4-97d255bbe6e4\") " pod="ai-tenant-e2e-shared-a-w2-98a394/rate-test-model-98a394-kserve-6d7884c4c7-gp5g4" Nov 03 02:33:35.615957 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:35.615844 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/786969b9-72c9-4b56-91e4-97d255bbe6e4-tmp-dir\") pod \"rate-test-model-98a394-kserve-6d7884c4c7-gp5g4\" (UID: \"786969b9-72c9-4b56-91e4-97d255bbe6e4\") " pod="ai-tenant-e2e-shared-a-w2-98a394/rate-test-model-98a394-kserve-6d7884c4c7-gp5g4" Nov 03 02:33:35.615957 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:35.615868 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/786969b9-72c9-4b56-91e4-97d255bbe6e4-dshm\") pod \"rate-test-model-98a394-kserve-6d7884c4c7-gp5g4\" (UID: \"786969b9-72c9-4b56-91e4-97d255bbe6e4\") " pod="ai-tenant-e2e-shared-a-w2-98a394/rate-test-model-98a394-kserve-6d7884c4c7-gp5g4" Nov 03 02:33:35.615957 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:35.615900 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/786969b9-72c9-4b56-91e4-97d255bbe6e4-model-cache\") pod \"rate-test-model-98a394-kserve-6d7884c4c7-gp5g4\" (UID: \"786969b9-72c9-4b56-91e4-97d255bbe6e4\") " pod="ai-tenant-e2e-shared-a-w2-98a394/rate-test-model-98a394-kserve-6d7884c4c7-gp5g4" Nov 03 02:33:35.615957 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:35.615936 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8gpr\" (UniqueName: \"kubernetes.io/projected/786969b9-72c9-4b56-91e4-97d255bbe6e4-kube-api-access-v8gpr\") pod \"rate-test-model-98a394-kserve-6d7884c4c7-gp5g4\" (UID: \"786969b9-72c9-4b56-91e4-97d255bbe6e4\") " pod="ai-tenant-e2e-shared-a-w2-98a394/rate-test-model-98a394-kserve-6d7884c4c7-gp5g4" Nov 03 02:33:35.616309 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:35.616234 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/786969b9-72c9-4b56-91e4-97d255bbe6e4-home\") pod \"rate-test-model-98a394-kserve-6d7884c4c7-gp5g4\" (UID: \"786969b9-72c9-4b56-91e4-97d255bbe6e4\") " pod="ai-tenant-e2e-shared-a-w2-98a394/rate-test-model-98a394-kserve-6d7884c4c7-gp5g4" Nov 03 02:33:35.616403 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:35.616345 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/786969b9-72c9-4b56-91e4-97d255bbe6e4-model-cache\") pod \"rate-test-model-98a394-kserve-6d7884c4c7-gp5g4\" (UID: \"786969b9-72c9-4b56-91e4-97d255bbe6e4\") " pod="ai-tenant-e2e-shared-a-w2-98a394/rate-test-model-98a394-kserve-6d7884c4c7-gp5g4" Nov 03 02:33:35.616604 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:35.616585 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/786969b9-72c9-4b56-91e4-97d255bbe6e4-tmp-dir\") pod \"rate-test-model-98a394-kserve-6d7884c4c7-gp5g4\" (UID: \"786969b9-72c9-4b56-91e4-97d255bbe6e4\") " pod="ai-tenant-e2e-shared-a-w2-98a394/rate-test-model-98a394-kserve-6d7884c4c7-gp5g4" Nov 03 02:33:35.618529 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:35.618489 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/786969b9-72c9-4b56-91e4-97d255bbe6e4-dshm\") pod \"rate-test-model-98a394-kserve-6d7884c4c7-gp5g4\" (UID: \"786969b9-72c9-4b56-91e4-97d255bbe6e4\") " pod="ai-tenant-e2e-shared-a-w2-98a394/rate-test-model-98a394-kserve-6d7884c4c7-gp5g4" Nov 03 02:33:35.619458 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:35.619400 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/786969b9-72c9-4b56-91e4-97d255bbe6e4-tls-certs\") pod \"rate-test-model-98a394-kserve-6d7884c4c7-gp5g4\" (UID: \"786969b9-72c9-4b56-91e4-97d255bbe6e4\") " pod="ai-tenant-e2e-shared-a-w2-98a394/rate-test-model-98a394-kserve-6d7884c4c7-gp5g4" Nov 03 02:33:35.679579 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:35.679493 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8gpr\" (UniqueName: \"kubernetes.io/projected/786969b9-72c9-4b56-91e4-97d255bbe6e4-kube-api-access-v8gpr\") pod \"rate-test-model-98a394-kserve-6d7884c4c7-gp5g4\" (UID: \"786969b9-72c9-4b56-91e4-97d255bbe6e4\") " pod="ai-tenant-e2e-shared-a-w2-98a394/rate-test-model-98a394-kserve-6d7884c4c7-gp5g4" Nov 03 02:33:35.740663 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:35.740626 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-98a394/rate-test-model-98a394-kserve-6d7884c4c7-gp5g4" Nov 03 02:33:36.115528 ip-10-0-136-16 kubenswrapper[5437]: W1103 02:33:36.115495 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod786969b9_72c9_4b56_91e4_97d255bbe6e4.slice/crio-6ae0482e8289602cfcef45b3298c8df2d7b1eb631c16737495842caca992ba2c WatchSource:0}: Error finding container 6ae0482e8289602cfcef45b3298c8df2d7b1eb631c16737495842caca992ba2c: Status 404 returned error can't find the container with id 6ae0482e8289602cfcef45b3298c8df2d7b1eb631c16737495842caca992ba2c Nov 03 02:33:36.118813 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:36.118786 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-98a394/rate-test-model-98a394-kserve-6d7884c4c7-gp5g4"] Nov 03 02:33:36.338758 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:36.338720 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-98a394/rate-test-model-98a394-kserve-6d7884c4c7-gp5g4" event={"ID":"786969b9-72c9-4b56-91e4-97d255bbe6e4","Type":"ContainerStarted","Data":"6b5fddaf6969ccbec71ef3f69e35af11ec74b017e065ac02ed99bff495aa6670"} Nov 03 02:33:36.338989 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:36.338774 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-98a394/rate-test-model-98a394-kserve-6d7884c4c7-gp5g4" event={"ID":"786969b9-72c9-4b56-91e4-97d255bbe6e4","Type":"ContainerStarted","Data":"6ae0482e8289602cfcef45b3298c8df2d7b1eb631c16737495842caca992ba2c"} Nov 03 02:33:36.460620 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:36.460509 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-98a394/rate-test-model-98a394-kserve-6d7884c4c7-gp5g4" podStartSLOduration=1.460492372 podStartE2EDuration="1.460492372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:33:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:33:36.458549937 +0000 UTC m=+4498.488032232" watchObservedRunningTime="2026-11-03 02:33:36.460492372 +0000 UTC m=+4498.489974668" Nov 03 02:33:36.741759 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:36.741664 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-98a394/rate-test-model-98a394-kserve-6d7884c4c7-gp5g4" Nov 03 02:33:37.708820 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:37.708783 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:33:39.159319 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:39.159275 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:33:39.358481 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:39.358390 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:33:39.566070 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:39.566032 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:33:41.307672 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:41.307642 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-eec2ff70-55d4b96df5-zzjsd" Nov 03 02:33:43.341732 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:43.341693 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a54cab-7b76bc44ff-9g8h8"] Nov 03 02:33:43.342633 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:43.342594 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a54cab-7b76bc44ff-9g8h8" podUID="f073cb08-11df-43ab-bb6a-79331e93ac08" containerName="maas-api" containerID="cri-o://0a5f608f765f2b3c98aad35b4b34d8ec9c37b6bc81b11734c74af5015640c1ba" gracePeriod=30 Nov 03 02:33:43.509337 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:43.509303 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:33:43.613734 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:43.613706 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a54cab-7b76bc44ff-9g8h8" Nov 03 02:33:43.716125 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:43.716086 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f073cb08-11df-43ab-bb6a-79331e93ac08-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f073cb08-11df-43ab-bb6a-79331e93ac08-metrics-certs\") pod \"f073cb08-11df-43ab-bb6a-79331e93ac08\" (UID: \"f073cb08-11df-43ab-bb6a-79331e93ac08\") " Nov 03 02:33:43.716304 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:43.716161 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f073cb08-11df-43ab-bb6a-79331e93ac08-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f073cb08-11df-43ab-bb6a-79331e93ac08-openshift-service-ca\") pod \"f073cb08-11df-43ab-bb6a-79331e93ac08\" (UID: \"f073cb08-11df-43ab-bb6a-79331e93ac08\") " Nov 03 02:33:43.716356 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:43.716311 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f073cb08-11df-43ab-bb6a-79331e93ac08-kube-api-access-2xngf\" (UniqueName: \"kubernetes.io/projected/f073cb08-11df-43ab-bb6a-79331e93ac08-kube-api-access-2xngf\") pod \"f073cb08-11df-43ab-bb6a-79331e93ac08\" (UID: \"f073cb08-11df-43ab-bb6a-79331e93ac08\") " Nov 03 02:33:43.716458 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:43.716435 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f073cb08-11df-43ab-bb6a-79331e93ac08-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f073cb08-11df-43ab-bb6a-79331e93ac08-maas-api-tls\") pod \"f073cb08-11df-43ab-bb6a-79331e93ac08\" (UID: \"f073cb08-11df-43ab-bb6a-79331e93ac08\") " Nov 03 02:33:43.716627 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:43.716606 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f073cb08-11df-43ab-bb6a-79331e93ac08-openshift-service-ca" pod "f073cb08-11df-43ab-bb6a-79331e93ac08" (UID: "f073cb08-11df-43ab-bb6a-79331e93ac08"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:33:43.717024 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:43.716971 5437 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f073cb08-11df-43ab-bb6a-79331e93ac08-openshift-service-ca\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:33:43.718671 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:43.718637 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f073cb08-11df-43ab-bb6a-79331e93ac08-maas-api-tls" pod "f073cb08-11df-43ab-bb6a-79331e93ac08" (UID: "f073cb08-11df-43ab-bb6a-79331e93ac08"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:33:43.718805 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:43.718741 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f073cb08-11df-43ab-bb6a-79331e93ac08-metrics-certs" pod "f073cb08-11df-43ab-bb6a-79331e93ac08" (UID: "f073cb08-11df-43ab-bb6a-79331e93ac08"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:33:43.722566 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:43.722539 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f073cb08-11df-43ab-bb6a-79331e93ac08-kube-api-access-2xngf" pod "f073cb08-11df-43ab-bb6a-79331e93ac08" (UID: "f073cb08-11df-43ab-bb6a-79331e93ac08"). InnerVolumeSpecName "kube-api-access-2xngf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:33:43.818198 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:43.818163 5437 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f073cb08-11df-43ab-bb6a-79331e93ac08-metrics-certs\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:33:43.818198 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:43.818194 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xngf\" (UniqueName: \"kubernetes.io/projected/f073cb08-11df-43ab-bb6a-79331e93ac08-kube-api-access-2xngf\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:33:43.818198 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:43.818204 5437 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f073cb08-11df-43ab-bb6a-79331e93ac08-maas-api-tls\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:33:44.035528 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:33:44.035437 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 02:33:44.390516 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:44.390483 5437 generic.go:356] "Generic (PLEG): container finished" podID="f073cb08-11df-43ab-bb6a-79331e93ac08" containerID="0a5f608f765f2b3c98aad35b4b34d8ec9c37b6bc81b11734c74af5015640c1ba" exitCode=0 Nov 03 02:33:44.391036 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:44.390539 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a54cab-7b76bc44ff-9g8h8" Nov 03 02:33:44.391036 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:44.390568 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a54cab-7b76bc44ff-9g8h8" event={"ID":"f073cb08-11df-43ab-bb6a-79331e93ac08","Type":"ContainerDied","Data":"0a5f608f765f2b3c98aad35b4b34d8ec9c37b6bc81b11734c74af5015640c1ba"} Nov 03 02:33:44.391036 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:44.390613 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a54cab-7b76bc44ff-9g8h8" event={"ID":"f073cb08-11df-43ab-bb6a-79331e93ac08","Type":"ContainerDied","Data":"68111bf0de62a89013c7f311c3541dd070454c9d1236e8cf2bc31462b16c108a"} Nov 03 02:33:44.391036 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:44.390639 5437 scope.go:122] "RemoveContainer" containerID="0a5f608f765f2b3c98aad35b4b34d8ec9c37b6bc81b11734c74af5015640c1ba" Nov 03 02:33:44.400834 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:44.400814 5437 scope.go:122] "RemoveContainer" containerID="0a5f608f765f2b3c98aad35b4b34d8ec9c37b6bc81b11734c74af5015640c1ba" Nov 03 02:33:44.401138 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:33:44.401114 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a5f608f765f2b3c98aad35b4b34d8ec9c37b6bc81b11734c74af5015640c1ba\": container with ID starting with 0a5f608f765f2b3c98aad35b4b34d8ec9c37b6bc81b11734c74af5015640c1ba not found: ID does not exist" containerID="0a5f608f765f2b3c98aad35b4b34d8ec9c37b6bc81b11734c74af5015640c1ba" Nov 03 02:33:44.401190 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:44.401148 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a5f608f765f2b3c98aad35b4b34d8ec9c37b6bc81b11734c74af5015640c1ba"} err="failed to get container status \"0a5f608f765f2b3c98aad35b4b34d8ec9c37b6bc81b11734c74af5015640c1ba\": rpc error: code = NotFound desc = could not find container \"0a5f608f765f2b3c98aad35b4b34d8ec9c37b6bc81b11734c74af5015640c1ba\": container with ID starting with 0a5f608f765f2b3c98aad35b4b34d8ec9c37b6bc81b11734c74af5015640c1ba not found: ID does not exist" Nov 03 02:33:44.561286 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:44.561254 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a54cab-7b76bc44ff-9g8h8"] Nov 03 02:33:44.612702 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:44.612671 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a54cab-7b76bc44ff-9g8h8"] Nov 03 02:33:45.149210 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:45.149168 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:33:45.740894 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:45.740862 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-98a394/rate-test-model-98a394-kserve-6d7884c4c7-gp5g4" Nov 03 02:33:45.758576 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:45.758542 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-98a394/rate-test-model-98a394-kserve-6d7884c4c7-gp5g4" Nov 03 02:33:46.089972 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:46.089939 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f073cb08-11df-43ab-bb6a-79331e93ac08" path="/var/lib/kubelet/pods/f073cb08-11df-43ab-bb6a-79331e93ac08/volumes" Nov 03 02:33:46.421386 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:46.421282 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-98a394/rate-test-model-98a394-kserve-6d7884c4c7-gp5g4" Nov 03 02:33:49.888013 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:49.887932 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:33:52.577747 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:52.577706 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:33:52.780819 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:52.780782 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:33:54.059621 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:54.059581 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-d9576a/rate-test-model-d9576a-kserve-779b5b8fc-lzrmc"] Nov 03 02:33:54.060440 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:54.060416 5437 cpu_manager.go:414] "containerMap: removing container" podUID="f073cb08-11df-43ab-bb6a-79331e93ac08" containerName="maas-api" Nov 03 02:33:54.060440 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:54.060440 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f073cb08-11df-43ab-bb6a-79331e93ac08" containerName="maas-api" Nov 03 02:33:54.060671 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:54.060656 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="f073cb08-11df-43ab-bb6a-79331e93ac08" containerName="maas-api" Nov 03 02:33:54.064003 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:54.063982 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-d9576a/rate-test-model-d9576a-kserve-779b5b8fc-lzrmc" Nov 03 02:33:54.085487 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:54.085448 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:33:54.119119 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:54.119087 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-d9576a/rate-test-model-d9576a-kserve-779b5b8fc-lzrmc"] Nov 03 02:33:54.168183 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:54.168142 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-d9576a\"/\"default-dockercfg-g8sfg\"" Nov 03 02:33:54.168183 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:54.168142 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-d9576a\"/\"kube-root-ca.crt\"" Nov 03 02:33:54.168183 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:54.168142 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-d9576a\"/\"openshift-service-ca.crt\"" Nov 03 02:33:54.170029 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:54.170015 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-d9576a\"/\"rate-test-model-d9576a-kserve-self-signed-certs\"" Nov 03 02:33:54.247013 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:54.246971 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1907086-a8a1-40a5-9a85-8b9d9ea21e40-model-cache\") pod \"rate-test-model-d9576a-kserve-779b5b8fc-lzrmc\" (UID: \"e1907086-a8a1-40a5-9a85-8b9d9ea21e40\") " pod="ai-tenant-e2e-shared-b-w4-d9576a/rate-test-model-d9576a-kserve-779b5b8fc-lzrmc" Nov 03 02:33:54.247194 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:54.247106 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1907086-a8a1-40a5-9a85-8b9d9ea21e40-tmp-dir\") pod \"rate-test-model-d9576a-kserve-779b5b8fc-lzrmc\" (UID: \"e1907086-a8a1-40a5-9a85-8b9d9ea21e40\") " pod="ai-tenant-e2e-shared-b-w4-d9576a/rate-test-model-d9576a-kserve-779b5b8fc-lzrmc" Nov 03 02:33:54.247240 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:54.247198 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lr9dw\" (UniqueName: \"kubernetes.io/projected/e1907086-a8a1-40a5-9a85-8b9d9ea21e40-kube-api-access-lr9dw\") pod \"rate-test-model-d9576a-kserve-779b5b8fc-lzrmc\" (UID: \"e1907086-a8a1-40a5-9a85-8b9d9ea21e40\") " pod="ai-tenant-e2e-shared-b-w4-d9576a/rate-test-model-d9576a-kserve-779b5b8fc-lzrmc" Nov 03 02:33:54.247278 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:54.247263 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1907086-a8a1-40a5-9a85-8b9d9ea21e40-home\") pod \"rate-test-model-d9576a-kserve-779b5b8fc-lzrmc\" (UID: \"e1907086-a8a1-40a5-9a85-8b9d9ea21e40\") " pod="ai-tenant-e2e-shared-b-w4-d9576a/rate-test-model-d9576a-kserve-779b5b8fc-lzrmc" Nov 03 02:33:54.247491 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:54.247461 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1907086-a8a1-40a5-9a85-8b9d9ea21e40-tls-certs\") pod \"rate-test-model-d9576a-kserve-779b5b8fc-lzrmc\" (UID: \"e1907086-a8a1-40a5-9a85-8b9d9ea21e40\") " pod="ai-tenant-e2e-shared-b-w4-d9576a/rate-test-model-d9576a-kserve-779b5b8fc-lzrmc" Nov 03 02:33:54.247563 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:54.247546 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1907086-a8a1-40a5-9a85-8b9d9ea21e40-dshm\") pod \"rate-test-model-d9576a-kserve-779b5b8fc-lzrmc\" (UID: \"e1907086-a8a1-40a5-9a85-8b9d9ea21e40\") " pod="ai-tenant-e2e-shared-b-w4-d9576a/rate-test-model-d9576a-kserve-779b5b8fc-lzrmc" Nov 03 02:33:54.336618 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:54.336515 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:33:54.349487 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:54.349457 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1907086-a8a1-40a5-9a85-8b9d9ea21e40-tls-certs\") pod \"rate-test-model-d9576a-kserve-779b5b8fc-lzrmc\" (UID: \"e1907086-a8a1-40a5-9a85-8b9d9ea21e40\") " pod="ai-tenant-e2e-shared-b-w4-d9576a/rate-test-model-d9576a-kserve-779b5b8fc-lzrmc" Nov 03 02:33:54.349650 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:54.349496 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1907086-a8a1-40a5-9a85-8b9d9ea21e40-dshm\") pod \"rate-test-model-d9576a-kserve-779b5b8fc-lzrmc\" (UID: \"e1907086-a8a1-40a5-9a85-8b9d9ea21e40\") " pod="ai-tenant-e2e-shared-b-w4-d9576a/rate-test-model-d9576a-kserve-779b5b8fc-lzrmc" Nov 03 02:33:54.349650 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:54.349561 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1907086-a8a1-40a5-9a85-8b9d9ea21e40-model-cache\") pod \"rate-test-model-d9576a-kserve-779b5b8fc-lzrmc\" (UID: \"e1907086-a8a1-40a5-9a85-8b9d9ea21e40\") " pod="ai-tenant-e2e-shared-b-w4-d9576a/rate-test-model-d9576a-kserve-779b5b8fc-lzrmc" Nov 03 02:33:54.349803 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:54.349720 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1907086-a8a1-40a5-9a85-8b9d9ea21e40-tmp-dir\") pod \"rate-test-model-d9576a-kserve-779b5b8fc-lzrmc\" (UID: \"e1907086-a8a1-40a5-9a85-8b9d9ea21e40\") " pod="ai-tenant-e2e-shared-b-w4-d9576a/rate-test-model-d9576a-kserve-779b5b8fc-lzrmc" Nov 03 02:33:54.349803 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:54.349780 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lr9dw\" (UniqueName: \"kubernetes.io/projected/e1907086-a8a1-40a5-9a85-8b9d9ea21e40-kube-api-access-lr9dw\") pod \"rate-test-model-d9576a-kserve-779b5b8fc-lzrmc\" (UID: \"e1907086-a8a1-40a5-9a85-8b9d9ea21e40\") " pod="ai-tenant-e2e-shared-b-w4-d9576a/rate-test-model-d9576a-kserve-779b5b8fc-lzrmc" Nov 03 02:33:54.349910 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:54.349820 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1907086-a8a1-40a5-9a85-8b9d9ea21e40-home\") pod \"rate-test-model-d9576a-kserve-779b5b8fc-lzrmc\" (UID: \"e1907086-a8a1-40a5-9a85-8b9d9ea21e40\") " pod="ai-tenant-e2e-shared-b-w4-d9576a/rate-test-model-d9576a-kserve-779b5b8fc-lzrmc" Nov 03 02:33:54.350172 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:54.350144 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1907086-a8a1-40a5-9a85-8b9d9ea21e40-model-cache\") pod \"rate-test-model-d9576a-kserve-779b5b8fc-lzrmc\" (UID: \"e1907086-a8a1-40a5-9a85-8b9d9ea21e40\") " pod="ai-tenant-e2e-shared-b-w4-d9576a/rate-test-model-d9576a-kserve-779b5b8fc-lzrmc" Nov 03 02:33:54.350428 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:54.350405 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1907086-a8a1-40a5-9a85-8b9d9ea21e40-home\") pod \"rate-test-model-d9576a-kserve-779b5b8fc-lzrmc\" (UID: \"e1907086-a8a1-40a5-9a85-8b9d9ea21e40\") " pod="ai-tenant-e2e-shared-b-w4-d9576a/rate-test-model-d9576a-kserve-779b5b8fc-lzrmc" Nov 03 02:33:54.350515 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:54.350438 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1907086-a8a1-40a5-9a85-8b9d9ea21e40-tmp-dir\") pod \"rate-test-model-d9576a-kserve-779b5b8fc-lzrmc\" (UID: \"e1907086-a8a1-40a5-9a85-8b9d9ea21e40\") " pod="ai-tenant-e2e-shared-b-w4-d9576a/rate-test-model-d9576a-kserve-779b5b8fc-lzrmc" Nov 03 02:33:54.352065 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:54.352038 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1907086-a8a1-40a5-9a85-8b9d9ea21e40-dshm\") pod \"rate-test-model-d9576a-kserve-779b5b8fc-lzrmc\" (UID: \"e1907086-a8a1-40a5-9a85-8b9d9ea21e40\") " pod="ai-tenant-e2e-shared-b-w4-d9576a/rate-test-model-d9576a-kserve-779b5b8fc-lzrmc" Nov 03 02:33:54.352279 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:54.352259 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1907086-a8a1-40a5-9a85-8b9d9ea21e40-tls-certs\") pod \"rate-test-model-d9576a-kserve-779b5b8fc-lzrmc\" (UID: \"e1907086-a8a1-40a5-9a85-8b9d9ea21e40\") " pod="ai-tenant-e2e-shared-b-w4-d9576a/rate-test-model-d9576a-kserve-779b5b8fc-lzrmc" Nov 03 02:33:54.414601 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:54.414566 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lr9dw\" (UniqueName: \"kubernetes.io/projected/e1907086-a8a1-40a5-9a85-8b9d9ea21e40-kube-api-access-lr9dw\") pod \"rate-test-model-d9576a-kserve-779b5b8fc-lzrmc\" (UID: \"e1907086-a8a1-40a5-9a85-8b9d9ea21e40\") " pod="ai-tenant-e2e-shared-b-w4-d9576a/rate-test-model-d9576a-kserve-779b5b8fc-lzrmc" Nov 03 02:33:54.568037 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:54.567996 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:33:54.683595 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:54.683504 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-d9576a/rate-test-model-d9576a-kserve-779b5b8fc-lzrmc" Nov 03 02:33:54.883607 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:54.883569 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:33:55.055288 ip-10-0-136-16 kubenswrapper[5437]: W1103 02:33:55.055259 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode1907086_a8a1_40a5_9a85_8b9d9ea21e40.slice/crio-0b67061ee7eb40f2febef7605e1fc6769ef3b7f8aa00857ec80c46c95763ca80 WatchSource:0}: Error finding container 0b67061ee7eb40f2febef7605e1fc6769ef3b7f8aa00857ec80c46c95763ca80: Status 404 returned error can't find the container with id 0b67061ee7eb40f2febef7605e1fc6769ef3b7f8aa00857ec80c46c95763ca80 Nov 03 02:33:55.058691 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:55.058667 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-d9576a/rate-test-model-d9576a-kserve-779b5b8fc-lzrmc"] Nov 03 02:33:55.349109 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:55.349013 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:33:55.468547 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:55.468504 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-d9576a/rate-test-model-d9576a-kserve-779b5b8fc-lzrmc" event={"ID":"e1907086-a8a1-40a5-9a85-8b9d9ea21e40","Type":"ContainerStarted","Data":"c55116364beede002ac238a51fed534cb9dbcb619c934c76ea6ec13cab0a8b66"} Nov 03 02:33:55.468725 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:55.468559 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-d9576a/rate-test-model-d9576a-kserve-779b5b8fc-lzrmc" event={"ID":"e1907086-a8a1-40a5-9a85-8b9d9ea21e40","Type":"ContainerStarted","Data":"0b67061ee7eb40f2febef7605e1fc6769ef3b7f8aa00857ec80c46c95763ca80"} Nov 03 02:33:55.588789 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:55.588739 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-d9576a/rate-test-model-d9576a-kserve-779b5b8fc-lzrmc" podStartSLOduration=1.588725119 podStartE2EDuration="1.588725119s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:33:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:33:55.587806045 +0000 UTC m=+4517.617288341" watchObservedRunningTime="2026-11-03 02:33:55.588725119 +0000 UTC m=+4517.618207414" Nov 03 02:33:55.684043 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:55.683952 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-d9576a/rate-test-model-d9576a-kserve-779b5b8fc-lzrmc" Nov 03 02:33:59.772868 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:33:59.772830 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:34:01.130750 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:34:01.123009 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:34:01.333161 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:34:01.333121 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:34:01.546731 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:34:01.546672 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:34:01.750325 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:34:01.750292 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:34:04.684319 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:34:04.684287 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-d9576a/rate-test-model-d9576a-kserve-779b5b8fc-lzrmc" Nov 03 02:34:04.697279 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:34:04.697252 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-d9576a/rate-test-model-d9576a-kserve-779b5b8fc-lzrmc" Nov 03 02:34:04.777035 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:34:04.776991 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:34:05.543306 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:34:05.543274 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-d9576a/rate-test-model-d9576a-kserve-779b5b8fc-lzrmc" Nov 03 02:34:06.491214 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:34:06.491170 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:34:07.477128 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:34:07.477093 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:34:09.488185 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:34:09.488141 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:34:14.034428 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:34:14.034387 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 02:34:14.394229 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:34:14.394191 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:34:19.094100 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:34:19.094057 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:34:23.891138 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:34:23.891102 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:34:28.780930 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:34:28.780889 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:34:33.422206 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:34:33.422165 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:34:33.652709 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:34:33.652670 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:34:33.855914 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:34:33.855873 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:34:38.497012 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:34:38.496971 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:34:43.186284 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:34:43.186237 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:34:44.035126 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:34:44.035088 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 02:34:47.894661 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:34:47.894612 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:34:52.768645 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:34:52.768597 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:34:52.975596 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:34:52.975554 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:34:57.473909 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:34:57.473857 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:34:59.525455 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:34:59.525400 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:34:59.970211 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:34:59.970172 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:35:00.584359 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:00.584320 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:35:01.153644 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:01.153602 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:35:01.878975 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:01.878937 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:35:02.569228 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:02.569178 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:35:05.724710 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:05.724669 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:35:05.933843 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:05.933809 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:35:06.823731 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:06.823699 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:35:07.141202 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:07.141108 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:35:08.867158 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:08.867117 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:35:09.113068 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:09.113013 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:35:13.800216 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:13.800176 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:35:14.034304 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:35:14.034264 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 02:35:15.569324 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:15.569282 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:35:15.785020 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:15.784983 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:35:17.076884 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:17.076799 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:35:17.589078 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:17.589043 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:35:18.092559 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:18.092527 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:35:18.793837 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:18.793761 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:35:22.996291 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:22.996256 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-e1793d/distinct-2-w1-e1793d-kserve-84fcdfb9fd-r27tq"] Nov 03 02:35:23.047406 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:23.047353 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-e1793d/distinct-w1-e1793d-kserve-687855c49-vn8zs"] Nov 03 02:35:23.049873 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:23.049841 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-e1793d/simulator-w1-e1793d-kserve-7595dfc6b7-dbbgf"] Nov 03 02:35:23.212182 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:23.212132 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:35:23.678877 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:23.678820 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:35:25.294855 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:25.294817 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:35:28.481701 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:28.481660 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:35:31.743584 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:31.743543 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:35:31.776244 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:31.776210 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-e1793d-7f578596cf-fj8br"] Nov 03 02:35:32.081022 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:32.080977 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:35:32.282133 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:32.282096 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:35:33.397121 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:33.397080 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:35:36.777146 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:36.777101 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e1793d-7f578596cf-fj8br" podUID="3029fee9-a1c7-4340-b52f-5fe182e44252" containerName="payload-pre-processing" containerID="cri-o://48a451ee9203ad15ae0cebeed7d9004998590005542499f27a85333f30a3bc52" gracePeriod=25 Nov 03 02:35:37.054552 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:37.054517 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e1793d-7f578596cf-fj8br" Nov 03 02:35:37.116899 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:37.116867 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3029fee9-a1c7-4340-b52f-5fe182e44252-kube-api-access-jjlwd\" (UniqueName: \"kubernetes.io/projected/3029fee9-a1c7-4340-b52f-5fe182e44252-kube-api-access-jjlwd\") pod \"3029fee9-a1c7-4340-b52f-5fe182e44252\" (UID: \"3029fee9-a1c7-4340-b52f-5fe182e44252\") " Nov 03 02:35:37.117107 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:37.116939 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3029fee9-a1c7-4340-b52f-5fe182e44252-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3029fee9-a1c7-4340-b52f-5fe182e44252-plugins-config-volume\") pod \"3029fee9-a1c7-4340-b52f-5fe182e44252\" (UID: \"3029fee9-a1c7-4340-b52f-5fe182e44252\") " Nov 03 02:35:37.117441 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:37.117407 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3029fee9-a1c7-4340-b52f-5fe182e44252-plugins-config-volume" pod "3029fee9-a1c7-4340-b52f-5fe182e44252" (UID: "3029fee9-a1c7-4340-b52f-5fe182e44252"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:35:37.117915 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:37.117888 5437 generic.go:356] "Generic (PLEG): container finished" podID="3029fee9-a1c7-4340-b52f-5fe182e44252" containerID="48a451ee9203ad15ae0cebeed7d9004998590005542499f27a85333f30a3bc52" exitCode=0 Nov 03 02:35:37.118041 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:37.117935 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e1793d-7f578596cf-fj8br" Nov 03 02:35:37.118041 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:37.117958 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e1793d-7f578596cf-fj8br" event={"ID":"3029fee9-a1c7-4340-b52f-5fe182e44252","Type":"ContainerDied","Data":"48a451ee9203ad15ae0cebeed7d9004998590005542499f27a85333f30a3bc52"} Nov 03 02:35:37.118041 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:37.118002 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e1793d-7f578596cf-fj8br" event={"ID":"3029fee9-a1c7-4340-b52f-5fe182e44252","Type":"ContainerDied","Data":"9be001cc7cee69c1b7ac35973a1711158cf505f7dda11bc41f9e8e416d1509da"} Nov 03 02:35:37.118041 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:37.118023 5437 scope.go:122] "RemoveContainer" containerID="48a451ee9203ad15ae0cebeed7d9004998590005542499f27a85333f30a3bc52" Nov 03 02:35:37.119456 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:37.119417 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3029fee9-a1c7-4340-b52f-5fe182e44252-kube-api-access-jjlwd" pod "3029fee9-a1c7-4340-b52f-5fe182e44252" (UID: "3029fee9-a1c7-4340-b52f-5fe182e44252"). InnerVolumeSpecName "kube-api-access-jjlwd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:35:37.135354 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:37.135333 5437 scope.go:122] "RemoveContainer" containerID="48a451ee9203ad15ae0cebeed7d9004998590005542499f27a85333f30a3bc52" Nov 03 02:35:37.135680 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:35:37.135657 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48a451ee9203ad15ae0cebeed7d9004998590005542499f27a85333f30a3bc52\": container with ID starting with 48a451ee9203ad15ae0cebeed7d9004998590005542499f27a85333f30a3bc52 not found: ID does not exist" containerID="48a451ee9203ad15ae0cebeed7d9004998590005542499f27a85333f30a3bc52" Nov 03 02:35:37.135745 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:37.135688 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48a451ee9203ad15ae0cebeed7d9004998590005542499f27a85333f30a3bc52"} err="failed to get container status \"48a451ee9203ad15ae0cebeed7d9004998590005542499f27a85333f30a3bc52\": rpc error: code = NotFound desc = could not find container \"48a451ee9203ad15ae0cebeed7d9004998590005542499f27a85333f30a3bc52\": container with ID starting with 48a451ee9203ad15ae0cebeed7d9004998590005542499f27a85333f30a3bc52 not found: ID does not exist" Nov 03 02:35:37.218346 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:37.218308 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjlwd\" (UniqueName: \"kubernetes.io/projected/3029fee9-a1c7-4340-b52f-5fe182e44252-kube-api-access-jjlwd\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:35:37.218346 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:37.218343 5437 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3029fee9-a1c7-4340-b52f-5fe182e44252-plugins-config-volume\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:35:37.549777 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:37.549739 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-e1793d-7f578596cf-fj8br"] Nov 03 02:35:37.552052 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:37.552027 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-e1793d-7f578596cf-fj8br"] Nov 03 02:35:37.785189 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:37.785095 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:35:38.024653 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:38.024604 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-e1793d/distinct-2-w1-e1793d-kserve-84fcdfb9fd-r27tq" podUID="2cf938f2-75d3-4af7-8645-cdf69a1d2744" containerName="main" containerID="cri-o://01d98296a853cda31ecc51480f7034e42dbd9e85f5a20127e8aeb160782efeb9" gracePeriod=45 Nov 03 02:35:38.067995 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:38.067949 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-e1793d/distinct-w1-e1793d-kserve-687855c49-vn8zs" podUID="a821ee29-70aa-4cad-87d4-3fb0edd20810" containerName="main" containerID="cri-o://1ae7738e8ab100813db6356624907d891521f65a24c45110960cd0a59c93a7fb" gracePeriod=45 Nov 03 02:35:38.069792 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:38.069746 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-e1793d/simulator-w1-e1793d-kserve-7595dfc6b7-dbbgf" podUID="f28ccf15-b92d-400c-9204-69b66279d7d8" containerName="main" containerID="cri-o://5bb9895276de4d8c7039a44493615e6ccf353375f7c04526e05b468626216b2d" gracePeriod=45 Nov 03 02:35:38.089050 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:38.089024 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3029fee9-a1c7-4340-b52f-5fe182e44252" path="/var/lib/kubelet/pods/3029fee9-a1c7-4340-b52f-5fe182e44252/volumes" Nov 03 02:35:38.114691 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:38.114666 5437 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e1793d/simulator-w1-e1793d-kserve-7595dfc6b7-dbbgf" podUID="f28ccf15-b92d-400c-9204-69b66279d7d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": dial tcp 10.134.0.66:8000: connect: connection refused" Nov 03 02:35:38.274263 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:38.274195 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:35:38.372986 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:38.372952 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e1793d/distinct-2-w1-e1793d-kserve-84fcdfb9fd-r27tq" Nov 03 02:35:38.394843 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:38.394813 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e1793d/simulator-w1-e1793d-kserve-7595dfc6b7-dbbgf" Nov 03 02:35:38.398752 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:38.398725 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e1793d/distinct-w1-e1793d-kserve-687855c49-vn8zs" Nov 03 02:35:38.432410 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:38.432374 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a821ee29-70aa-4cad-87d4-3fb0edd20810-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a821ee29-70aa-4cad-87d4-3fb0edd20810-dshm\") pod \"a821ee29-70aa-4cad-87d4-3fb0edd20810\" (UID: \"a821ee29-70aa-4cad-87d4-3fb0edd20810\") " Nov 03 02:35:38.432577 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:38.432443 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a821ee29-70aa-4cad-87d4-3fb0edd20810-kube-api-access-r4qrg\" (UniqueName: \"kubernetes.io/projected/a821ee29-70aa-4cad-87d4-3fb0edd20810-kube-api-access-r4qrg\") pod \"a821ee29-70aa-4cad-87d4-3fb0edd20810\" (UID: \"a821ee29-70aa-4cad-87d4-3fb0edd20810\") " Nov 03 02:35:38.432577 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:38.432476 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f28ccf15-b92d-400c-9204-69b66279d7d8-tls-certs\" (UniqueName: \"kubernetes.io/secret/f28ccf15-b92d-400c-9204-69b66279d7d8-tls-certs\") pod \"f28ccf15-b92d-400c-9204-69b66279d7d8\" (UID: \"f28ccf15-b92d-400c-9204-69b66279d7d8\") " Nov 03 02:35:38.432577 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:38.432519 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a821ee29-70aa-4cad-87d4-3fb0edd20810-home\" (UniqueName: \"kubernetes.io/empty-dir/a821ee29-70aa-4cad-87d4-3fb0edd20810-home\") pod \"a821ee29-70aa-4cad-87d4-3fb0edd20810\" (UID: \"a821ee29-70aa-4cad-87d4-3fb0edd20810\") " Nov 03 02:35:38.432754 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:38.432598 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f28ccf15-b92d-400c-9204-69b66279d7d8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f28ccf15-b92d-400c-9204-69b66279d7d8-model-cache\") pod \"f28ccf15-b92d-400c-9204-69b66279d7d8\" (UID: \"f28ccf15-b92d-400c-9204-69b66279d7d8\") " Nov 03 02:35:38.432754 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:38.432645 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f28ccf15-b92d-400c-9204-69b66279d7d8-home\" (UniqueName: \"kubernetes.io/empty-dir/f28ccf15-b92d-400c-9204-69b66279d7d8-home\") pod \"f28ccf15-b92d-400c-9204-69b66279d7d8\" (UID: \"f28ccf15-b92d-400c-9204-69b66279d7d8\") " Nov 03 02:35:38.432754 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:38.432687 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cf938f2-75d3-4af7-8645-cdf69a1d2744-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cf938f2-75d3-4af7-8645-cdf69a1d2744-model-cache\") pod \"2cf938f2-75d3-4af7-8645-cdf69a1d2744\" (UID: \"2cf938f2-75d3-4af7-8645-cdf69a1d2744\") " Nov 03 02:35:38.432924 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:38.432754 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a821ee29-70aa-4cad-87d4-3fb0edd20810-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a821ee29-70aa-4cad-87d4-3fb0edd20810-model-cache\") pod \"a821ee29-70aa-4cad-87d4-3fb0edd20810\" (UID: \"a821ee29-70aa-4cad-87d4-3fb0edd20810\") " Nov 03 02:35:38.432924 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:38.432795 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a821ee29-70aa-4cad-87d4-3fb0edd20810-home" pod "a821ee29-70aa-4cad-87d4-3fb0edd20810" (UID: "a821ee29-70aa-4cad-87d4-3fb0edd20810"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:35:38.432924 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:38.432817 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2cf938f2-75d3-4af7-8645-cdf69a1d2744-tls-certs\" (UniqueName: \"kubernetes.io/secret/2cf938f2-75d3-4af7-8645-cdf69a1d2744-tls-certs\") pod \"2cf938f2-75d3-4af7-8645-cdf69a1d2744\" (UID: \"2cf938f2-75d3-4af7-8645-cdf69a1d2744\") " Nov 03 02:35:38.432924 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:38.432867 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cf938f2-75d3-4af7-8645-cdf69a1d2744-home\" (UniqueName: \"kubernetes.io/empty-dir/2cf938f2-75d3-4af7-8645-cdf69a1d2744-home\") pod \"2cf938f2-75d3-4af7-8645-cdf69a1d2744\" (UID: \"2cf938f2-75d3-4af7-8645-cdf69a1d2744\") " Nov 03 02:35:38.432924 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:38.432910 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f28ccf15-b92d-400c-9204-69b66279d7d8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f28ccf15-b92d-400c-9204-69b66279d7d8-dshm\") pod \"f28ccf15-b92d-400c-9204-69b66279d7d8\" (UID: \"f28ccf15-b92d-400c-9204-69b66279d7d8\") " Nov 03 02:35:38.432924 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:38.432913 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f28ccf15-b92d-400c-9204-69b66279d7d8-model-cache" pod "f28ccf15-b92d-400c-9204-69b66279d7d8" (UID: "f28ccf15-b92d-400c-9204-69b66279d7d8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:35:38.433256 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:38.432949 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cf938f2-75d3-4af7-8645-cdf69a1d2744-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2cf938f2-75d3-4af7-8645-cdf69a1d2744-dshm\") pod \"2cf938f2-75d3-4af7-8645-cdf69a1d2744\" (UID: \"2cf938f2-75d3-4af7-8645-cdf69a1d2744\") " Nov 03 02:35:38.433256 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:38.432960 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f28ccf15-b92d-400c-9204-69b66279d7d8-home" pod "f28ccf15-b92d-400c-9204-69b66279d7d8" (UID: "f28ccf15-b92d-400c-9204-69b66279d7d8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:35:38.433256 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:38.432987 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f28ccf15-b92d-400c-9204-69b66279d7d8-kube-api-access-rtfnh\" (UniqueName: \"kubernetes.io/projected/f28ccf15-b92d-400c-9204-69b66279d7d8-kube-api-access-rtfnh\") pod \"f28ccf15-b92d-400c-9204-69b66279d7d8\" (UID: \"f28ccf15-b92d-400c-9204-69b66279d7d8\") " Nov 03 02:35:38.433256 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:38.433001 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cf938f2-75d3-4af7-8645-cdf69a1d2744-model-cache" pod "2cf938f2-75d3-4af7-8645-cdf69a1d2744" (UID: "2cf938f2-75d3-4af7-8645-cdf69a1d2744"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:35:38.433256 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:38.433033 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a821ee29-70aa-4cad-87d4-3fb0edd20810-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a821ee29-70aa-4cad-87d4-3fb0edd20810-tmp-dir\") pod \"a821ee29-70aa-4cad-87d4-3fb0edd20810\" (UID: \"a821ee29-70aa-4cad-87d4-3fb0edd20810\") " Nov 03 02:35:38.433256 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:38.433064 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cf938f2-75d3-4af7-8645-cdf69a1d2744-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cf938f2-75d3-4af7-8645-cdf69a1d2744-tmp-dir\") pod \"2cf938f2-75d3-4af7-8645-cdf69a1d2744\" (UID: \"2cf938f2-75d3-4af7-8645-cdf69a1d2744\") " Nov 03 02:35:38.433256 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:38.433104 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f28ccf15-b92d-400c-9204-69b66279d7d8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f28ccf15-b92d-400c-9204-69b66279d7d8-tmp-dir\") pod \"f28ccf15-b92d-400c-9204-69b66279d7d8\" (UID: \"f28ccf15-b92d-400c-9204-69b66279d7d8\") " Nov 03 02:35:38.433256 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:38.433176 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a821ee29-70aa-4cad-87d4-3fb0edd20810-model-cache" pod "a821ee29-70aa-4cad-87d4-3fb0edd20810" (UID: "a821ee29-70aa-4cad-87d4-3fb0edd20810"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:35:38.433256 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:38.433184 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a821ee29-70aa-4cad-87d4-3fb0edd20810-tls-certs\" (UniqueName: \"kubernetes.io/secret/a821ee29-70aa-4cad-87d4-3fb0edd20810-tls-certs\") pod \"a821ee29-70aa-4cad-87d4-3fb0edd20810\" (UID: \"a821ee29-70aa-4cad-87d4-3fb0edd20810\") " Nov 03 02:35:38.433256 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:38.433212 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cf938f2-75d3-4af7-8645-cdf69a1d2744-kube-api-access-6v8rs\" (UniqueName: \"kubernetes.io/projected/2cf938f2-75d3-4af7-8645-cdf69a1d2744-kube-api-access-6v8rs\") pod \"2cf938f2-75d3-4af7-8645-cdf69a1d2744\" (UID: \"2cf938f2-75d3-4af7-8645-cdf69a1d2744\") " Nov 03 02:35:38.433837 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:38.433386 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a821ee29-70aa-4cad-87d4-3fb0edd20810-tmp-dir" pod "a821ee29-70aa-4cad-87d4-3fb0edd20810" (UID: "a821ee29-70aa-4cad-87d4-3fb0edd20810"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:35:38.433837 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:38.433826 5437 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f28ccf15-b92d-400c-9204-69b66279d7d8-model-cache\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:35:38.433963 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:38.433848 5437 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f28ccf15-b92d-400c-9204-69b66279d7d8-home\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:35:38.433963 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:38.433864 5437 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cf938f2-75d3-4af7-8645-cdf69a1d2744-model-cache\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:35:38.433963 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:38.433879 5437 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a821ee29-70aa-4cad-87d4-3fb0edd20810-model-cache\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:35:38.433963 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:38.433893 5437 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a821ee29-70aa-4cad-87d4-3fb0edd20810-tmp-dir\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:35:38.433963 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:38.433905 5437 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a821ee29-70aa-4cad-87d4-3fb0edd20810-home\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:35:38.434764 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:38.434493 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cf938f2-75d3-4af7-8645-cdf69a1d2744-home" pod "2cf938f2-75d3-4af7-8645-cdf69a1d2744" (UID: "2cf938f2-75d3-4af7-8645-cdf69a1d2744"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:35:38.435639 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:38.435410 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f28ccf15-b92d-400c-9204-69b66279d7d8-tls-certs" pod "f28ccf15-b92d-400c-9204-69b66279d7d8" (UID: "f28ccf15-b92d-400c-9204-69b66279d7d8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:35:38.435639 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:38.435606 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a821ee29-70aa-4cad-87d4-3fb0edd20810-kube-api-access-r4qrg" pod "a821ee29-70aa-4cad-87d4-3fb0edd20810" (UID: "a821ee29-70aa-4cad-87d4-3fb0edd20810"). InnerVolumeSpecName "kube-api-access-r4qrg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:35:38.435819 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:38.435657 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f28ccf15-b92d-400c-9204-69b66279d7d8-dshm" pod "f28ccf15-b92d-400c-9204-69b66279d7d8" (UID: "f28ccf15-b92d-400c-9204-69b66279d7d8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:35:38.435819 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:38.435805 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cf938f2-75d3-4af7-8645-cdf69a1d2744-tmp-dir" pod "2cf938f2-75d3-4af7-8645-cdf69a1d2744" (UID: "2cf938f2-75d3-4af7-8645-cdf69a1d2744"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:35:38.435996 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:38.435953 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cf938f2-75d3-4af7-8645-cdf69a1d2744-kube-api-access-6v8rs" pod "2cf938f2-75d3-4af7-8645-cdf69a1d2744" (UID: "2cf938f2-75d3-4af7-8645-cdf69a1d2744"). InnerVolumeSpecName "kube-api-access-6v8rs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:35:38.435996 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:38.435972 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f28ccf15-b92d-400c-9204-69b66279d7d8-tmp-dir" pod "f28ccf15-b92d-400c-9204-69b66279d7d8" (UID: "f28ccf15-b92d-400c-9204-69b66279d7d8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:35:38.436123 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:38.435995 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2cf938f2-75d3-4af7-8645-cdf69a1d2744-tls-certs" pod "2cf938f2-75d3-4af7-8645-cdf69a1d2744" (UID: "2cf938f2-75d3-4af7-8645-cdf69a1d2744"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:35:38.436269 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:38.436251 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cf938f2-75d3-4af7-8645-cdf69a1d2744-dshm" pod "2cf938f2-75d3-4af7-8645-cdf69a1d2744" (UID: "2cf938f2-75d3-4af7-8645-cdf69a1d2744"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:35:38.436834 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:38.436811 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f28ccf15-b92d-400c-9204-69b66279d7d8-kube-api-access-rtfnh" pod "f28ccf15-b92d-400c-9204-69b66279d7d8" (UID: "f28ccf15-b92d-400c-9204-69b66279d7d8"). InnerVolumeSpecName "kube-api-access-rtfnh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:35:38.436902 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:38.436878 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a821ee29-70aa-4cad-87d4-3fb0edd20810-dshm" pod "a821ee29-70aa-4cad-87d4-3fb0edd20810" (UID: "a821ee29-70aa-4cad-87d4-3fb0edd20810"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:35:38.437698 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:38.437682 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a821ee29-70aa-4cad-87d4-3fb0edd20810-tls-certs" pod "a821ee29-70aa-4cad-87d4-3fb0edd20810" (UID: "a821ee29-70aa-4cad-87d4-3fb0edd20810"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:35:38.535651 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:38.535612 5437 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a821ee29-70aa-4cad-87d4-3fb0edd20810-tls-certs\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:35:38.535651 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:38.535652 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6v8rs\" (UniqueName: \"kubernetes.io/projected/2cf938f2-75d3-4af7-8645-cdf69a1d2744-kube-api-access-6v8rs\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:35:38.535651 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:38.535668 5437 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a821ee29-70aa-4cad-87d4-3fb0edd20810-dshm\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:35:38.536129 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:38.535682 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4qrg\" (UniqueName: \"kubernetes.io/projected/a821ee29-70aa-4cad-87d4-3fb0edd20810-kube-api-access-r4qrg\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:35:38.536129 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:38.535698 5437 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f28ccf15-b92d-400c-9204-69b66279d7d8-tls-certs\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:35:38.536129 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:38.535712 5437 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2cf938f2-75d3-4af7-8645-cdf69a1d2744-tls-certs\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:35:38.536129 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:38.535724 5437 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2cf938f2-75d3-4af7-8645-cdf69a1d2744-home\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:35:38.536129 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:38.535736 5437 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f28ccf15-b92d-400c-9204-69b66279d7d8-dshm\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:35:38.536129 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:38.535750 5437 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2cf938f2-75d3-4af7-8645-cdf69a1d2744-dshm\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:35:38.536129 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:38.535763 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtfnh\" (UniqueName: \"kubernetes.io/projected/f28ccf15-b92d-400c-9204-69b66279d7d8-kube-api-access-rtfnh\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:35:38.536129 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:38.535778 5437 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cf938f2-75d3-4af7-8645-cdf69a1d2744-tmp-dir\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:35:38.536129 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:38.535791 5437 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f28ccf15-b92d-400c-9204-69b66279d7d8-tmp-dir\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:35:39.132130 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:39.132094 5437 generic.go:356] "Generic (PLEG): container finished" podID="f28ccf15-b92d-400c-9204-69b66279d7d8" containerID="5bb9895276de4d8c7039a44493615e6ccf353375f7c04526e05b468626216b2d" exitCode=0 Nov 03 02:35:39.132591 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:39.132143 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e1793d/simulator-w1-e1793d-kserve-7595dfc6b7-dbbgf" Nov 03 02:35:39.132591 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:39.132186 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e1793d/simulator-w1-e1793d-kserve-7595dfc6b7-dbbgf" event={"ID":"f28ccf15-b92d-400c-9204-69b66279d7d8","Type":"ContainerDied","Data":"5bb9895276de4d8c7039a44493615e6ccf353375f7c04526e05b468626216b2d"} Nov 03 02:35:39.132591 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:39.132231 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e1793d/simulator-w1-e1793d-kserve-7595dfc6b7-dbbgf" event={"ID":"f28ccf15-b92d-400c-9204-69b66279d7d8","Type":"ContainerDied","Data":"8559ea3d503bd3c5ae2926cdedfa394367971561b4ba3a150e69bcc4c7c5ded1"} Nov 03 02:35:39.132591 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:39.132253 5437 scope.go:122] "RemoveContainer" containerID="5bb9895276de4d8c7039a44493615e6ccf353375f7c04526e05b468626216b2d" Nov 03 02:35:39.133932 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:39.133907 5437 generic.go:356] "Generic (PLEG): container finished" podID="a821ee29-70aa-4cad-87d4-3fb0edd20810" containerID="1ae7738e8ab100813db6356624907d891521f65a24c45110960cd0a59c93a7fb" exitCode=0 Nov 03 02:35:39.134043 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:39.133940 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e1793d/distinct-w1-e1793d-kserve-687855c49-vn8zs" Nov 03 02:35:39.134043 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:39.133977 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e1793d/distinct-w1-e1793d-kserve-687855c49-vn8zs" event={"ID":"a821ee29-70aa-4cad-87d4-3fb0edd20810","Type":"ContainerDied","Data":"1ae7738e8ab100813db6356624907d891521f65a24c45110960cd0a59c93a7fb"} Nov 03 02:35:39.134043 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:39.134010 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e1793d/distinct-w1-e1793d-kserve-687855c49-vn8zs" event={"ID":"a821ee29-70aa-4cad-87d4-3fb0edd20810","Type":"ContainerDied","Data":"12ee961752075d9ffa31c587c0fe9feba341083be1b1fb5967e8a73c7c7d3ec0"} Nov 03 02:35:39.135793 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:39.135771 5437 generic.go:356] "Generic (PLEG): container finished" podID="2cf938f2-75d3-4af7-8645-cdf69a1d2744" containerID="01d98296a853cda31ecc51480f7034e42dbd9e85f5a20127e8aeb160782efeb9" exitCode=0 Nov 03 02:35:39.135899 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:39.135811 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e1793d/distinct-2-w1-e1793d-kserve-84fcdfb9fd-r27tq" event={"ID":"2cf938f2-75d3-4af7-8645-cdf69a1d2744","Type":"ContainerDied","Data":"01d98296a853cda31ecc51480f7034e42dbd9e85f5a20127e8aeb160782efeb9"} Nov 03 02:35:39.135899 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:39.135823 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e1793d/distinct-2-w1-e1793d-kserve-84fcdfb9fd-r27tq" Nov 03 02:35:39.135899 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:39.135832 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e1793d/distinct-2-w1-e1793d-kserve-84fcdfb9fd-r27tq" event={"ID":"2cf938f2-75d3-4af7-8645-cdf69a1d2744","Type":"ContainerDied","Data":"a280bfe8b3a2c47a92dcccf12e40cba3571d53ca688f505136fcd50fc81a250f"} Nov 03 02:35:39.143181 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:39.143157 5437 scope.go:122] "RemoveContainer" containerID="5bb9895276de4d8c7039a44493615e6ccf353375f7c04526e05b468626216b2d" Nov 03 02:35:39.143497 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:35:39.143460 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bb9895276de4d8c7039a44493615e6ccf353375f7c04526e05b468626216b2d\": container with ID starting with 5bb9895276de4d8c7039a44493615e6ccf353375f7c04526e05b468626216b2d not found: ID does not exist" containerID="5bb9895276de4d8c7039a44493615e6ccf353375f7c04526e05b468626216b2d" Nov 03 02:35:39.143596 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:39.143506 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bb9895276de4d8c7039a44493615e6ccf353375f7c04526e05b468626216b2d"} err="failed to get container status \"5bb9895276de4d8c7039a44493615e6ccf353375f7c04526e05b468626216b2d\": rpc error: code = NotFound desc = could not find container \"5bb9895276de4d8c7039a44493615e6ccf353375f7c04526e05b468626216b2d\": container with ID starting with 5bb9895276de4d8c7039a44493615e6ccf353375f7c04526e05b468626216b2d not found: ID does not exist" Nov 03 02:35:39.143596 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:39.143529 5437 scope.go:122] "RemoveContainer" containerID="1ae7738e8ab100813db6356624907d891521f65a24c45110960cd0a59c93a7fb" Nov 03 02:35:39.152926 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:39.152906 5437 scope.go:122] "RemoveContainer" containerID="1ae7738e8ab100813db6356624907d891521f65a24c45110960cd0a59c93a7fb" Nov 03 02:35:39.153202 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:35:39.153184 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ae7738e8ab100813db6356624907d891521f65a24c45110960cd0a59c93a7fb\": container with ID starting with 1ae7738e8ab100813db6356624907d891521f65a24c45110960cd0a59c93a7fb not found: ID does not exist" containerID="1ae7738e8ab100813db6356624907d891521f65a24c45110960cd0a59c93a7fb" Nov 03 02:35:39.153260 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:39.153208 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ae7738e8ab100813db6356624907d891521f65a24c45110960cd0a59c93a7fb"} err="failed to get container status \"1ae7738e8ab100813db6356624907d891521f65a24c45110960cd0a59c93a7fb\": rpc error: code = NotFound desc = could not find container \"1ae7738e8ab100813db6356624907d891521f65a24c45110960cd0a59c93a7fb\": container with ID starting with 1ae7738e8ab100813db6356624907d891521f65a24c45110960cd0a59c93a7fb not found: ID does not exist" Nov 03 02:35:39.153260 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:39.153226 5437 scope.go:122] "RemoveContainer" containerID="01d98296a853cda31ecc51480f7034e42dbd9e85f5a20127e8aeb160782efeb9" Nov 03 02:35:39.164115 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:39.164094 5437 scope.go:122] "RemoveContainer" containerID="01d98296a853cda31ecc51480f7034e42dbd9e85f5a20127e8aeb160782efeb9" Nov 03 02:35:39.164625 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:35:39.164600 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01d98296a853cda31ecc51480f7034e42dbd9e85f5a20127e8aeb160782efeb9\": container with ID starting with 01d98296a853cda31ecc51480f7034e42dbd9e85f5a20127e8aeb160782efeb9 not found: ID does not exist" containerID="01d98296a853cda31ecc51480f7034e42dbd9e85f5a20127e8aeb160782efeb9" Nov 03 02:35:39.164718 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:39.164632 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01d98296a853cda31ecc51480f7034e42dbd9e85f5a20127e8aeb160782efeb9"} err="failed to get container status \"01d98296a853cda31ecc51480f7034e42dbd9e85f5a20127e8aeb160782efeb9\": rpc error: code = NotFound desc = could not find container \"01d98296a853cda31ecc51480f7034e42dbd9e85f5a20127e8aeb160782efeb9\": container with ID starting with 01d98296a853cda31ecc51480f7034e42dbd9e85f5a20127e8aeb160782efeb9 not found: ID does not exist" Nov 03 02:35:39.302229 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:39.302186 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-e1793d/simulator-w1-e1793d-kserve-7595dfc6b7-dbbgf"] Nov 03 02:35:39.338910 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:39.338870 5437 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e1793d/distinct-w1-e1793d-kserve-687855c49-vn8zs" podUID="a821ee29-70aa-4cad-87d4-3fb0edd20810" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 02:35:39.354092 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:39.354060 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-e1793d/simulator-w1-e1793d-kserve-7595dfc6b7-dbbgf"] Nov 03 02:35:39.419051 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:39.419018 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-e1793d/distinct-w1-e1793d-kserve-687855c49-vn8zs"] Nov 03 02:35:39.424178 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:39.424152 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-e1793d/distinct-w1-e1793d-kserve-687855c49-vn8zs"] Nov 03 02:35:39.537396 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:39.537354 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-e1793d/distinct-2-w1-e1793d-kserve-84fcdfb9fd-r27tq"] Nov 03 02:35:39.539913 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:39.539888 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-e1793d/distinct-2-w1-e1793d-kserve-84fcdfb9fd-r27tq"] Nov 03 02:35:40.089615 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:40.089583 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2cf938f2-75d3-4af7-8645-cdf69a1d2744" path="/var/lib/kubelet/pods/2cf938f2-75d3-4af7-8645-cdf69a1d2744/volumes" Nov 03 02:35:40.090050 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:40.090037 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a821ee29-70aa-4cad-87d4-3fb0edd20810" path="/var/lib/kubelet/pods/a821ee29-70aa-4cad-87d4-3fb0edd20810/volumes" Nov 03 02:35:40.090446 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:40.090432 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f28ccf15-b92d-400c-9204-69b66279d7d8" path="/var/lib/kubelet/pods/f28ccf15-b92d-400c-9204-69b66279d7d8/volumes" Nov 03 02:35:43.075252 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:43.075214 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:35:43.495776 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:43.495663 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:35:44.035424 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:35:44.035359 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 02:35:47.526806 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:47.526751 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:35:47.866341 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:47.866293 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:35:48.986891 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:48.986853 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:35:49.819901 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:49.819857 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:35:50.079842 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:50.079746 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:35:50.440324 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:50.440233 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:35:53.082843 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:53.082807 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:35:55.825828 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:55.825792 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mg5rw"] Nov 03 02:35:55.826398 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:55.826382 5437 cpu_manager.go:414] "containerMap: removing container" podUID="f28ccf15-b92d-400c-9204-69b66279d7d8" containerName="main" Nov 03 02:35:55.826466 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:55.826398 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f28ccf15-b92d-400c-9204-69b66279d7d8" containerName="main" Nov 03 02:35:55.826466 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:55.826418 5437 cpu_manager.go:414] "containerMap: removing container" podUID="3029fee9-a1c7-4340-b52f-5fe182e44252" containerName="payload-pre-processing" Nov 03 02:35:55.826466 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:55.826424 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3029fee9-a1c7-4340-b52f-5fe182e44252" containerName="payload-pre-processing" Nov 03 02:35:55.826466 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:55.826432 5437 cpu_manager.go:414] "containerMap: removing container" podUID="a821ee29-70aa-4cad-87d4-3fb0edd20810" containerName="main" Nov 03 02:35:55.826466 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:55.826437 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a821ee29-70aa-4cad-87d4-3fb0edd20810" containerName="main" Nov 03 02:35:55.826466 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:55.826452 5437 cpu_manager.go:414] "containerMap: removing container" podUID="2cf938f2-75d3-4af7-8645-cdf69a1d2744" containerName="main" Nov 03 02:35:55.826466 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:55.826457 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cf938f2-75d3-4af7-8645-cdf69a1d2744" containerName="main" Nov 03 02:35:55.826718 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:55.826544 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cf938f2-75d3-4af7-8645-cdf69a1d2744" containerName="main" Nov 03 02:35:55.826718 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:55.826559 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="f28ccf15-b92d-400c-9204-69b66279d7d8" containerName="main" Nov 03 02:35:55.826718 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:55.826567 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="3029fee9-a1c7-4340-b52f-5fe182e44252" containerName="payload-pre-processing" Nov 03 02:35:55.826718 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:55.826572 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="a821ee29-70aa-4cad-87d4-3fb0edd20810" containerName="main" Nov 03 02:35:55.830025 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:55.830008 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mg5rw" Nov 03 02:35:55.884151 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:55.884120 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-e1793d-openshift-default-dockercfg-4jbqs\"" Nov 03 02:35:55.938285 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:55.938244 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mg5rw"] Nov 03 02:35:55.950356 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:55.950320 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87d1198d-17a2-4290-95b3-8e5d68fe17aa-istio-envoy\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mg5rw\" (UID: \"87d1198d-17a2-4290-95b3-8e5d68fe17aa\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mg5rw" Nov 03 02:35:55.950601 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:55.950434 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87d1198d-17a2-4290-95b3-8e5d68fe17aa-workload-certs\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mg5rw\" (UID: \"87d1198d-17a2-4290-95b3-8e5d68fe17aa\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mg5rw" Nov 03 02:35:55.950601 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:55.950553 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87d1198d-17a2-4290-95b3-8e5d68fe17aa-credential-socket\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mg5rw\" (UID: \"87d1198d-17a2-4290-95b3-8e5d68fe17aa\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mg5rw" Nov 03 02:35:55.950601 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:55.950579 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvfg5\" (UniqueName: \"kubernetes.io/projected/87d1198d-17a2-4290-95b3-8e5d68fe17aa-kube-api-access-rvfg5\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mg5rw\" (UID: \"87d1198d-17a2-4290-95b3-8e5d68fe17aa\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mg5rw" Nov 03 02:35:55.950793 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:55.950603 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87d1198d-17a2-4290-95b3-8e5d68fe17aa-istio-podinfo\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mg5rw\" (UID: \"87d1198d-17a2-4290-95b3-8e5d68fe17aa\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mg5rw" Nov 03 02:35:55.950793 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:55.950620 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87d1198d-17a2-4290-95b3-8e5d68fe17aa-istio-data\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mg5rw\" (UID: \"87d1198d-17a2-4290-95b3-8e5d68fe17aa\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mg5rw" Nov 03 02:35:55.950902 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:55.950789 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87d1198d-17a2-4290-95b3-8e5d68fe17aa-istiod-ca-cert\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mg5rw\" (UID: \"87d1198d-17a2-4290-95b3-8e5d68fe17aa\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mg5rw" Nov 03 02:35:55.950902 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:55.950835 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87d1198d-17a2-4290-95b3-8e5d68fe17aa-istio-token\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mg5rw\" (UID: \"87d1198d-17a2-4290-95b3-8e5d68fe17aa\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mg5rw" Nov 03 02:35:55.951020 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:55.950945 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87d1198d-17a2-4290-95b3-8e5d68fe17aa-workload-socket\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mg5rw\" (UID: \"87d1198d-17a2-4290-95b3-8e5d68fe17aa\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mg5rw" Nov 03 02:35:56.052596 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:56.052551 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87d1198d-17a2-4290-95b3-8e5d68fe17aa-istiod-ca-cert\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mg5rw\" (UID: \"87d1198d-17a2-4290-95b3-8e5d68fe17aa\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mg5rw" Nov 03 02:35:56.052772 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:56.052618 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87d1198d-17a2-4290-95b3-8e5d68fe17aa-istio-token\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mg5rw\" (UID: \"87d1198d-17a2-4290-95b3-8e5d68fe17aa\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mg5rw" Nov 03 02:35:56.052772 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:56.052680 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87d1198d-17a2-4290-95b3-8e5d68fe17aa-workload-socket\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mg5rw\" (UID: \"87d1198d-17a2-4290-95b3-8e5d68fe17aa\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mg5rw" Nov 03 02:35:56.052772 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:56.052760 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87d1198d-17a2-4290-95b3-8e5d68fe17aa-istio-envoy\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mg5rw\" (UID: \"87d1198d-17a2-4290-95b3-8e5d68fe17aa\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mg5rw" Nov 03 02:35:56.052969 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:56.052797 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87d1198d-17a2-4290-95b3-8e5d68fe17aa-workload-certs\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mg5rw\" (UID: \"87d1198d-17a2-4290-95b3-8e5d68fe17aa\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mg5rw" Nov 03 02:35:56.052969 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:56.052871 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87d1198d-17a2-4290-95b3-8e5d68fe17aa-credential-socket\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mg5rw\" (UID: \"87d1198d-17a2-4290-95b3-8e5d68fe17aa\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mg5rw" Nov 03 02:35:56.052969 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:56.052891 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvfg5\" (UniqueName: \"kubernetes.io/projected/87d1198d-17a2-4290-95b3-8e5d68fe17aa-kube-api-access-rvfg5\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mg5rw\" (UID: \"87d1198d-17a2-4290-95b3-8e5d68fe17aa\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mg5rw" Nov 03 02:35:56.052969 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:56.052917 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87d1198d-17a2-4290-95b3-8e5d68fe17aa-istio-podinfo\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mg5rw\" (UID: \"87d1198d-17a2-4290-95b3-8e5d68fe17aa\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mg5rw" Nov 03 02:35:56.052969 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:56.052932 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87d1198d-17a2-4290-95b3-8e5d68fe17aa-istio-data\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mg5rw\" (UID: \"87d1198d-17a2-4290-95b3-8e5d68fe17aa\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mg5rw" Nov 03 02:35:56.053225 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:56.053049 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87d1198d-17a2-4290-95b3-8e5d68fe17aa-workload-socket\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mg5rw\" (UID: \"87d1198d-17a2-4290-95b3-8e5d68fe17aa\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mg5rw" Nov 03 02:35:56.053321 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:56.053295 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87d1198d-17a2-4290-95b3-8e5d68fe17aa-credential-socket\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mg5rw\" (UID: \"87d1198d-17a2-4290-95b3-8e5d68fe17aa\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mg5rw" Nov 03 02:35:56.053414 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:56.053356 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87d1198d-17a2-4290-95b3-8e5d68fe17aa-istio-data\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mg5rw\" (UID: \"87d1198d-17a2-4290-95b3-8e5d68fe17aa\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mg5rw" Nov 03 02:35:56.053414 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:56.053395 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87d1198d-17a2-4290-95b3-8e5d68fe17aa-istiod-ca-cert\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mg5rw\" (UID: \"87d1198d-17a2-4290-95b3-8e5d68fe17aa\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mg5rw" Nov 03 02:35:56.053541 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:56.053467 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87d1198d-17a2-4290-95b3-8e5d68fe17aa-workload-certs\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mg5rw\" (UID: \"87d1198d-17a2-4290-95b3-8e5d68fe17aa\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mg5rw" Nov 03 02:35:56.055078 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:56.055051 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87d1198d-17a2-4290-95b3-8e5d68fe17aa-istio-envoy\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mg5rw\" (UID: \"87d1198d-17a2-4290-95b3-8e5d68fe17aa\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mg5rw" Nov 03 02:35:56.055525 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:56.055504 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87d1198d-17a2-4290-95b3-8e5d68fe17aa-istio-podinfo\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mg5rw\" (UID: \"87d1198d-17a2-4290-95b3-8e5d68fe17aa\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mg5rw" Nov 03 02:35:56.115534 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:56.115452 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87d1198d-17a2-4290-95b3-8e5d68fe17aa-istio-token\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mg5rw\" (UID: \"87d1198d-17a2-4290-95b3-8e5d68fe17aa\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mg5rw" Nov 03 02:35:56.115696 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:56.115672 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvfg5\" (UniqueName: \"kubernetes.io/projected/87d1198d-17a2-4290-95b3-8e5d68fe17aa-kube-api-access-rvfg5\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mg5rw\" (UID: \"87d1198d-17a2-4290-95b3-8e5d68fe17aa\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mg5rw" Nov 03 02:35:56.144202 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:56.144175 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mg5rw" Nov 03 02:35:56.566199 ip-10-0-136-16 kubenswrapper[5437]: W1103 02:35:56.566167 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87d1198d_17a2_4290_95b3_8e5d68fe17aa.slice/crio-68c062808c7881ade9f1ed1a7b31c2a4793a4c83b30676cb528294650b06802b WatchSource:0}: Error finding container 68c062808c7881ade9f1ed1a7b31c2a4793a4c83b30676cb528294650b06802b: Status 404 returned error can't find the container with id 68c062808c7881ade9f1ed1a7b31c2a4793a4c83b30676cb528294650b06802b Nov 03 02:35:56.569172 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:56.569133 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 02:35:56.569274 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:56.569219 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 02:35:56.569274 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:56.569266 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 02:35:56.569681 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:56.569660 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mg5rw"] Nov 03 02:35:57.111112 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:57.111076 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:35:57.246403 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:57.246356 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mg5rw" event={"ID":"87d1198d-17a2-4290-95b3-8e5d68fe17aa","Type":"ContainerStarted","Data":"891fcac2647761ebf1c48d7fc9b889a71dd292fcde031d3319b921df6f265e3a"} Nov 03 02:35:57.246581 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:57.246418 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mg5rw" event={"ID":"87d1198d-17a2-4290-95b3-8e5d68fe17aa","Type":"ContainerStarted","Data":"68c062808c7881ade9f1ed1a7b31c2a4793a4c83b30676cb528294650b06802b"} Nov 03 02:35:57.373273 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:57.373167 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mg5rw" podStartSLOduration=2.373148053 podStartE2EDuration="2.373148053s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:35:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:35:57.369845628 +0000 UTC m=+4639.399327924" watchObservedRunningTime="2026-11-03 02:35:57.373148053 +0000 UTC m=+4639.402630348" Nov 03 02:35:58.079632 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:58.079595 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:35:58.144438 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:58.144394 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mg5rw" Nov 03 02:35:58.146276 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:58.146248 5437 patch_prober.go:29] interesting pod/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mg5rw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" start-of-body= Nov 03 02:35:58.146431 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:58.146300 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mg5rw" podUID="87d1198d-17a2-4290-95b3-8e5d68fe17aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" Nov 03 02:35:59.145375 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:59.145332 5437 patch_prober.go:29] interesting pod/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mg5rw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" start-of-body= Nov 03 02:35:59.145786 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:35:59.145431 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mg5rw" podUID="87d1198d-17a2-4290-95b3-8e5d68fe17aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" Nov 03 02:36:00.144945 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:36:00.144904 5437 patch_prober.go:29] interesting pod/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mg5rw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" start-of-body= Nov 03 02:36:00.145230 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:36:00.144963 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mg5rw" podUID="87d1198d-17a2-4290-95b3-8e5d68fe17aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" Nov 03 02:36:00.512855 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:36:00.512740 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:36:00.759168 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:36:00.759125 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:36:01.044137 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:36:01.044086 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:36:01.145246 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:36:01.145205 5437 patch_prober.go:29] interesting pod/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mg5rw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" start-of-body= Nov 03 02:36:01.145448 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:36:01.145273 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mg5rw" podUID="87d1198d-17a2-4290-95b3-8e5d68fe17aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" Nov 03 02:36:02.144634 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:36:02.144601 5437 patch_prober.go:29] interesting pod/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mg5rw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" start-of-body= Nov 03 02:36:02.145097 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:36:02.144660 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mg5rw" podUID="87d1198d-17a2-4290-95b3-8e5d68fe17aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" Nov 03 02:36:02.227412 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:36:02.226094 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:36:03.046093 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:36:03.046042 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:36:03.144585 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:36:03.144549 5437 patch_prober.go:29] interesting pod/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mg5rw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" start-of-body= Nov 03 02:36:03.144779 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:36:03.144618 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mg5rw" podUID="87d1198d-17a2-4290-95b3-8e5d68fe17aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" Nov 03 02:36:04.393272 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:36:04.393230 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mg5rw" Nov 03 02:36:04.394159 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:36:04.394133 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mg5rw" Nov 03 02:36:04.395146 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:36:04.395121 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mg5rw" Nov 03 02:36:07.091469 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:36:07.091425 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:36:07.306965 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:36:07.306928 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:36:07.774186 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:36:07.774144 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:36:09.668692 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:36:09.668647 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:36:11.931531 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:36:11.931489 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:36:13.559617 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:36:13.559582 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:36:13.819007 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:36:13.818907 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:36:14.034682 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:36:14.034646 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 02:36:16.724958 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:36:16.724910 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:36:16.930057 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:36:16.930015 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:36:18.386692 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:36:18.386653 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:36:18.712031 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:36:18.711883 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:36:18.847173 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:36:18.847136 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mg5rw"] Nov 03 02:36:18.847384 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:36:18.847336 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mg5rw" podUID="87d1198d-17a2-4290-95b3-8e5d68fe17aa" containerName="istio-proxy" containerID="cri-o://891fcac2647761ebf1c48d7fc9b889a71dd292fcde031d3319b921df6f265e3a" gracePeriod=30 Nov 03 02:36:18.976555 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:36:18.976442 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:36:19.396676 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:36:19.396642 5437 patch_prober.go:29] interesting pod/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mg5rw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 02:36:19.397177 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:36:19.396707 5437 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mg5rw" podUID="87d1198d-17a2-4290-95b3-8e5d68fe17aa" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 02:36:21.449332 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:36:21.449293 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:36:21.703913 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:36:21.703812 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:36:23.702609 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:36:23.702570 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:36:24.116035 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:36:24.116006 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mg5rw" Nov 03 02:36:24.308772 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:36:24.308737 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/87d1198d-17a2-4290-95b3-8e5d68fe17aa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87d1198d-17a2-4290-95b3-8e5d68fe17aa-istio-podinfo\") pod \"87d1198d-17a2-4290-95b3-8e5d68fe17aa\" (UID: \"87d1198d-17a2-4290-95b3-8e5d68fe17aa\") " Nov 03 02:36:24.308979 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:36:24.308887 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87d1198d-17a2-4290-95b3-8e5d68fe17aa-kube-api-access-rvfg5\" (UniqueName: \"kubernetes.io/projected/87d1198d-17a2-4290-95b3-8e5d68fe17aa-kube-api-access-rvfg5\") pod \"87d1198d-17a2-4290-95b3-8e5d68fe17aa\" (UID: \"87d1198d-17a2-4290-95b3-8e5d68fe17aa\") " Nov 03 02:36:24.308979 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:36:24.308936 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87d1198d-17a2-4290-95b3-8e5d68fe17aa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87d1198d-17a2-4290-95b3-8e5d68fe17aa-workload-socket\") pod \"87d1198d-17a2-4290-95b3-8e5d68fe17aa\" (UID: \"87d1198d-17a2-4290-95b3-8e5d68fe17aa\") " Nov 03 02:36:24.309112 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:36:24.309001 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87d1198d-17a2-4290-95b3-8e5d68fe17aa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87d1198d-17a2-4290-95b3-8e5d68fe17aa-istiod-ca-cert\") pod \"87d1198d-17a2-4290-95b3-8e5d68fe17aa\" (UID: \"87d1198d-17a2-4290-95b3-8e5d68fe17aa\") " Nov 03 02:36:24.309112 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:36:24.309056 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87d1198d-17a2-4290-95b3-8e5d68fe17aa-istio-token\" (UniqueName: \"kubernetes.io/projected/87d1198d-17a2-4290-95b3-8e5d68fe17aa-istio-token\") pod \"87d1198d-17a2-4290-95b3-8e5d68fe17aa\" (UID: \"87d1198d-17a2-4290-95b3-8e5d68fe17aa\") " Nov 03 02:36:24.309309 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:36:24.309265 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87d1198d-17a2-4290-95b3-8e5d68fe17aa-workload-socket" pod "87d1198d-17a2-4290-95b3-8e5d68fe17aa" (UID: "87d1198d-17a2-4290-95b3-8e5d68fe17aa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:36:24.309400 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:36:24.309291 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87d1198d-17a2-4290-95b3-8e5d68fe17aa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87d1198d-17a2-4290-95b3-8e5d68fe17aa-credential-socket\") pod \"87d1198d-17a2-4290-95b3-8e5d68fe17aa\" (UID: \"87d1198d-17a2-4290-95b3-8e5d68fe17aa\") " Nov 03 02:36:24.309466 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:36:24.309396 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87d1198d-17a2-4290-95b3-8e5d68fe17aa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87d1198d-17a2-4290-95b3-8e5d68fe17aa-istio-envoy\") pod \"87d1198d-17a2-4290-95b3-8e5d68fe17aa\" (UID: \"87d1198d-17a2-4290-95b3-8e5d68fe17aa\") " Nov 03 02:36:24.309527 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:36:24.309503 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87d1198d-17a2-4290-95b3-8e5d68fe17aa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87d1198d-17a2-4290-95b3-8e5d68fe17aa-istio-data\") pod \"87d1198d-17a2-4290-95b3-8e5d68fe17aa\" (UID: \"87d1198d-17a2-4290-95b3-8e5d68fe17aa\") " Nov 03 02:36:24.309587 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:36:24.309536 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87d1198d-17a2-4290-95b3-8e5d68fe17aa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87d1198d-17a2-4290-95b3-8e5d68fe17aa-workload-certs\") pod \"87d1198d-17a2-4290-95b3-8e5d68fe17aa\" (UID: \"87d1198d-17a2-4290-95b3-8e5d68fe17aa\") " Nov 03 02:36:24.309642 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:36:24.309603 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87d1198d-17a2-4290-95b3-8e5d68fe17aa-credential-socket" pod "87d1198d-17a2-4290-95b3-8e5d68fe17aa" (UID: "87d1198d-17a2-4290-95b3-8e5d68fe17aa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:36:24.310110 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:36:24.310062 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87d1198d-17a2-4290-95b3-8e5d68fe17aa-istiod-ca-cert" pod "87d1198d-17a2-4290-95b3-8e5d68fe17aa" (UID: "87d1198d-17a2-4290-95b3-8e5d68fe17aa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:36:24.310110 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:36:24.310092 5437 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87d1198d-17a2-4290-95b3-8e5d68fe17aa-workload-socket\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:36:24.310110 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:36:24.310111 5437 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87d1198d-17a2-4290-95b3-8e5d68fe17aa-credential-socket\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:36:24.310289 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:36:24.310131 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87d1198d-17a2-4290-95b3-8e5d68fe17aa-workload-certs" pod "87d1198d-17a2-4290-95b3-8e5d68fe17aa" (UID: "87d1198d-17a2-4290-95b3-8e5d68fe17aa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:36:24.310289 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:36:24.310195 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87d1198d-17a2-4290-95b3-8e5d68fe17aa-istio-data" pod "87d1198d-17a2-4290-95b3-8e5d68fe17aa" (UID: "87d1198d-17a2-4290-95b3-8e5d68fe17aa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:36:24.311593 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:36:24.311561 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/87d1198d-17a2-4290-95b3-8e5d68fe17aa-istio-podinfo" pod "87d1198d-17a2-4290-95b3-8e5d68fe17aa" (UID: "87d1198d-17a2-4290-95b3-8e5d68fe17aa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:36:24.311764 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:36:24.311738 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87d1198d-17a2-4290-95b3-8e5d68fe17aa-istio-token" pod "87d1198d-17a2-4290-95b3-8e5d68fe17aa" (UID: "87d1198d-17a2-4290-95b3-8e5d68fe17aa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:36:24.311854 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:36:24.311832 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87d1198d-17a2-4290-95b3-8e5d68fe17aa-istio-envoy" pod "87d1198d-17a2-4290-95b3-8e5d68fe17aa" (UID: "87d1198d-17a2-4290-95b3-8e5d68fe17aa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:36:24.312096 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:36:24.312077 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87d1198d-17a2-4290-95b3-8e5d68fe17aa-kube-api-access-rvfg5" pod "87d1198d-17a2-4290-95b3-8e5d68fe17aa" (UID: "87d1198d-17a2-4290-95b3-8e5d68fe17aa"). InnerVolumeSpecName "kube-api-access-rvfg5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:36:24.411462 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:36:24.411431 5437 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87d1198d-17a2-4290-95b3-8e5d68fe17aa-istio-envoy\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:36:24.411462 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:36:24.411458 5437 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87d1198d-17a2-4290-95b3-8e5d68fe17aa-istio-data\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:36:24.411462 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:36:24.411474 5437 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87d1198d-17a2-4290-95b3-8e5d68fe17aa-workload-certs\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:36:24.411791 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:36:24.411486 5437 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87d1198d-17a2-4290-95b3-8e5d68fe17aa-istio-podinfo\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:36:24.411791 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:36:24.411495 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvfg5\" (UniqueName: \"kubernetes.io/projected/87d1198d-17a2-4290-95b3-8e5d68fe17aa-kube-api-access-rvfg5\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:36:24.411791 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:36:24.411504 5437 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87d1198d-17a2-4290-95b3-8e5d68fe17aa-istiod-ca-cert\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:36:24.411791 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:36:24.411514 5437 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87d1198d-17a2-4290-95b3-8e5d68fe17aa-istio-token\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:36:24.426240 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:36:24.426197 5437 generic.go:356] "Generic (PLEG): container finished" podID="87d1198d-17a2-4290-95b3-8e5d68fe17aa" containerID="891fcac2647761ebf1c48d7fc9b889a71dd292fcde031d3319b921df6f265e3a" exitCode=0 Nov 03 02:36:24.426431 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:36:24.426250 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mg5rw" Nov 03 02:36:24.426431 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:36:24.426283 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mg5rw" event={"ID":"87d1198d-17a2-4290-95b3-8e5d68fe17aa","Type":"ContainerDied","Data":"891fcac2647761ebf1c48d7fc9b889a71dd292fcde031d3319b921df6f265e3a"} Nov 03 02:36:24.426431 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:36:24.426334 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mg5rw" event={"ID":"87d1198d-17a2-4290-95b3-8e5d68fe17aa","Type":"ContainerDied","Data":"68c062808c7881ade9f1ed1a7b31c2a4793a4c83b30676cb528294650b06802b"} Nov 03 02:36:24.426431 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:36:24.426359 5437 scope.go:122] "RemoveContainer" containerID="891fcac2647761ebf1c48d7fc9b889a71dd292fcde031d3319b921df6f265e3a" Nov 03 02:36:24.437403 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:36:24.437379 5437 scope.go:122] "RemoveContainer" containerID="891fcac2647761ebf1c48d7fc9b889a71dd292fcde031d3319b921df6f265e3a" Nov 03 02:36:24.437716 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:36:24.437697 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"891fcac2647761ebf1c48d7fc9b889a71dd292fcde031d3319b921df6f265e3a\": container with ID starting with 891fcac2647761ebf1c48d7fc9b889a71dd292fcde031d3319b921df6f265e3a not found: ID does not exist" containerID="891fcac2647761ebf1c48d7fc9b889a71dd292fcde031d3319b921df6f265e3a" Nov 03 02:36:24.437765 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:36:24.437725 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"891fcac2647761ebf1c48d7fc9b889a71dd292fcde031d3319b921df6f265e3a"} err="failed to get container status \"891fcac2647761ebf1c48d7fc9b889a71dd292fcde031d3319b921df6f265e3a\": rpc error: code = NotFound desc = could not find container \"891fcac2647761ebf1c48d7fc9b889a71dd292fcde031d3319b921df6f265e3a\": container with ID starting with 891fcac2647761ebf1c48d7fc9b889a71dd292fcde031d3319b921df6f265e3a not found: ID does not exist" Nov 03 02:36:24.599829 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:36:24.599742 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mg5rw"] Nov 03 02:36:24.604903 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:36:24.604878 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-mg5rw"] Nov 03 02:36:26.089524 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:36:26.089488 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87d1198d-17a2-4290-95b3-8e5d68fe17aa" path="/var/lib/kubelet/pods/87d1198d-17a2-4290-95b3-8e5d68fe17aa/volumes" Nov 03 02:36:28.568663 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:36:28.568610 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:36:33.373616 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:36:33.373577 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:36:38.171187 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:36:38.171145 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:36:42.884293 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:36:42.884249 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:36:44.035051 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:36:44.035014 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 02:36:47.628623 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:36:47.628536 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:36:52.383145 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:36:52.383105 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:36:57.183100 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:36:57.183062 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:37:01.976785 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:37:01.976745 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:37:02.568182 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:37:02.568147 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:37:06.716254 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:37:06.716214 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:37:07.525481 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:37:07.525442 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:37:11.510496 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:37:11.510460 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:37:14.035938 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:37:14.035898 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 02:37:16.277965 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:37:16.277920 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:37:21.047513 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:37:21.047464 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:37:21.296434 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:37:21.296395 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:37:25.884626 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:37:25.884581 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:37:27.698975 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:37:27.698938 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:37:28.701400 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:37:28.701348 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:37:30.842718 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:37:30.842682 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-eec2ff70-55d4b96df5-zzjsd"] Nov 03 02:37:30.975106 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:37:30.975067 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:37:31.246158 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:37:31.246065 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:37:31.453193 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:37:31.453155 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:37:33.139450 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:37:33.139400 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w0-a3dce7/sub-test-model-a3dce7-kserve-78f7fc75f-pkntl"] Nov 03 02:37:33.140256 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:37:33.140232 5437 cpu_manager.go:414] "containerMap: removing container" podUID="87d1198d-17a2-4290-95b3-8e5d68fe17aa" containerName="istio-proxy" Nov 03 02:37:33.140391 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:37:33.140257 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87d1198d-17a2-4290-95b3-8e5d68fe17aa" containerName="istio-proxy" Nov 03 02:37:33.140521 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:37:33.140503 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="87d1198d-17a2-4290-95b3-8e5d68fe17aa" containerName="istio-proxy" Nov 03 02:37:33.143876 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:37:33.143854 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-a3dce7/sub-test-model-a3dce7-kserve-78f7fc75f-pkntl" Nov 03 02:37:33.197951 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:37:33.197918 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-a3dce7/sub-test-model-a3dce7-kserve-78f7fc75f-pkntl"] Nov 03 02:37:33.226495 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:37:33.226460 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cac18c4-e538-4c4b-85cf-de232f64de18-tmp-dir\") pod \"sub-test-model-a3dce7-kserve-78f7fc75f-pkntl\" (UID: \"2cac18c4-e538-4c4b-85cf-de232f64de18\") " pod="ai-tenant-e2e-shared-a-w0-a3dce7/sub-test-model-a3dce7-kserve-78f7fc75f-pkntl" Nov 03 02:37:33.226641 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:37:33.226613 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2cac18c4-e538-4c4b-85cf-de232f64de18-home\") pod \"sub-test-model-a3dce7-kserve-78f7fc75f-pkntl\" (UID: \"2cac18c4-e538-4c4b-85cf-de232f64de18\") " pod="ai-tenant-e2e-shared-a-w0-a3dce7/sub-test-model-a3dce7-kserve-78f7fc75f-pkntl" Nov 03 02:37:33.226896 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:37:33.226872 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dn5lb\" (UniqueName: \"kubernetes.io/projected/2cac18c4-e538-4c4b-85cf-de232f64de18-kube-api-access-dn5lb\") pod \"sub-test-model-a3dce7-kserve-78f7fc75f-pkntl\" (UID: \"2cac18c4-e538-4c4b-85cf-de232f64de18\") " pod="ai-tenant-e2e-shared-a-w0-a3dce7/sub-test-model-a3dce7-kserve-78f7fc75f-pkntl" Nov 03 02:37:33.227054 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:37:33.227035 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cac18c4-e538-4c4b-85cf-de232f64de18-model-cache\") pod \"sub-test-model-a3dce7-kserve-78f7fc75f-pkntl\" (UID: \"2cac18c4-e538-4c4b-85cf-de232f64de18\") " pod="ai-tenant-e2e-shared-a-w0-a3dce7/sub-test-model-a3dce7-kserve-78f7fc75f-pkntl" Nov 03 02:37:33.227138 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:37:33.227122 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2cac18c4-e538-4c4b-85cf-de232f64de18-dshm\") pod \"sub-test-model-a3dce7-kserve-78f7fc75f-pkntl\" (UID: \"2cac18c4-e538-4c4b-85cf-de232f64de18\") " pod="ai-tenant-e2e-shared-a-w0-a3dce7/sub-test-model-a3dce7-kserve-78f7fc75f-pkntl" Nov 03 02:37:33.227211 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:37:33.227192 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2cac18c4-e538-4c4b-85cf-de232f64de18-tls-certs\") pod \"sub-test-model-a3dce7-kserve-78f7fc75f-pkntl\" (UID: \"2cac18c4-e538-4c4b-85cf-de232f64de18\") " pod="ai-tenant-e2e-shared-a-w0-a3dce7/sub-test-model-a3dce7-kserve-78f7fc75f-pkntl" Nov 03 02:37:33.246315 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:37:33.246288 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-a3dce7\"/\"kube-root-ca.crt\"" Nov 03 02:37:33.246499 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:37:33.246420 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-a3dce7\"/\"sub-test-model-a3dce7-kserve-self-signed-certs\"" Nov 03 02:37:33.249579 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:37:33.249546 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-a3dce7\"/\"openshift-service-ca.crt\"" Nov 03 02:37:33.249672 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:37:33.249581 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-a3dce7\"/\"default-dockercfg-5d7lq\"" Nov 03 02:37:33.295290 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:37:33.295254 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:37:33.329066 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:37:33.329024 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cac18c4-e538-4c4b-85cf-de232f64de18-model-cache\") pod \"sub-test-model-a3dce7-kserve-78f7fc75f-pkntl\" (UID: \"2cac18c4-e538-4c4b-85cf-de232f64de18\") " pod="ai-tenant-e2e-shared-a-w0-a3dce7/sub-test-model-a3dce7-kserve-78f7fc75f-pkntl" Nov 03 02:37:33.329284 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:37:33.329113 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2cac18c4-e538-4c4b-85cf-de232f64de18-dshm\") pod \"sub-test-model-a3dce7-kserve-78f7fc75f-pkntl\" (UID: \"2cac18c4-e538-4c4b-85cf-de232f64de18\") " pod="ai-tenant-e2e-shared-a-w0-a3dce7/sub-test-model-a3dce7-kserve-78f7fc75f-pkntl" Nov 03 02:37:33.329284 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:37:33.329145 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2cac18c4-e538-4c4b-85cf-de232f64de18-tls-certs\") pod \"sub-test-model-a3dce7-kserve-78f7fc75f-pkntl\" (UID: \"2cac18c4-e538-4c4b-85cf-de232f64de18\") " pod="ai-tenant-e2e-shared-a-w0-a3dce7/sub-test-model-a3dce7-kserve-78f7fc75f-pkntl" Nov 03 02:37:33.329284 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:37:33.329211 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cac18c4-e538-4c4b-85cf-de232f64de18-tmp-dir\") pod \"sub-test-model-a3dce7-kserve-78f7fc75f-pkntl\" (UID: \"2cac18c4-e538-4c4b-85cf-de232f64de18\") " pod="ai-tenant-e2e-shared-a-w0-a3dce7/sub-test-model-a3dce7-kserve-78f7fc75f-pkntl" Nov 03 02:37:33.329284 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:37:33.329263 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2cac18c4-e538-4c4b-85cf-de232f64de18-home\") pod \"sub-test-model-a3dce7-kserve-78f7fc75f-pkntl\" (UID: \"2cac18c4-e538-4c4b-85cf-de232f64de18\") " pod="ai-tenant-e2e-shared-a-w0-a3dce7/sub-test-model-a3dce7-kserve-78f7fc75f-pkntl" Nov 03 02:37:33.329546 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:37:33.329338 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dn5lb\" (UniqueName: \"kubernetes.io/projected/2cac18c4-e538-4c4b-85cf-de232f64de18-kube-api-access-dn5lb\") pod \"sub-test-model-a3dce7-kserve-78f7fc75f-pkntl\" (UID: \"2cac18c4-e538-4c4b-85cf-de232f64de18\") " pod="ai-tenant-e2e-shared-a-w0-a3dce7/sub-test-model-a3dce7-kserve-78f7fc75f-pkntl" Nov 03 02:37:33.330152 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:37:33.330123 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cac18c4-e538-4c4b-85cf-de232f64de18-model-cache\") pod \"sub-test-model-a3dce7-kserve-78f7fc75f-pkntl\" (UID: \"2cac18c4-e538-4c4b-85cf-de232f64de18\") " pod="ai-tenant-e2e-shared-a-w0-a3dce7/sub-test-model-a3dce7-kserve-78f7fc75f-pkntl" Nov 03 02:37:33.330749 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:37:33.330719 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cac18c4-e538-4c4b-85cf-de232f64de18-tmp-dir\") pod \"sub-test-model-a3dce7-kserve-78f7fc75f-pkntl\" (UID: \"2cac18c4-e538-4c4b-85cf-de232f64de18\") " pod="ai-tenant-e2e-shared-a-w0-a3dce7/sub-test-model-a3dce7-kserve-78f7fc75f-pkntl" Nov 03 02:37:33.330905 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:37:33.330795 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2cac18c4-e538-4c4b-85cf-de232f64de18-home\") pod \"sub-test-model-a3dce7-kserve-78f7fc75f-pkntl\" (UID: \"2cac18c4-e538-4c4b-85cf-de232f64de18\") " pod="ai-tenant-e2e-shared-a-w0-a3dce7/sub-test-model-a3dce7-kserve-78f7fc75f-pkntl" Nov 03 02:37:33.333630 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:37:33.333603 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2cac18c4-e538-4c4b-85cf-de232f64de18-dshm\") pod \"sub-test-model-a3dce7-kserve-78f7fc75f-pkntl\" (UID: \"2cac18c4-e538-4c4b-85cf-de232f64de18\") " pod="ai-tenant-e2e-shared-a-w0-a3dce7/sub-test-model-a3dce7-kserve-78f7fc75f-pkntl" Nov 03 02:37:33.342248 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:37:33.342214 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2cac18c4-e538-4c4b-85cf-de232f64de18-tls-certs\") pod \"sub-test-model-a3dce7-kserve-78f7fc75f-pkntl\" (UID: \"2cac18c4-e538-4c4b-85cf-de232f64de18\") " pod="ai-tenant-e2e-shared-a-w0-a3dce7/sub-test-model-a3dce7-kserve-78f7fc75f-pkntl" Nov 03 02:37:33.392344 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:37:33.392267 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dn5lb\" (UniqueName: \"kubernetes.io/projected/2cac18c4-e538-4c4b-85cf-de232f64de18-kube-api-access-dn5lb\") pod \"sub-test-model-a3dce7-kserve-78f7fc75f-pkntl\" (UID: \"2cac18c4-e538-4c4b-85cf-de232f64de18\") " pod="ai-tenant-e2e-shared-a-w0-a3dce7/sub-test-model-a3dce7-kserve-78f7fc75f-pkntl" Nov 03 02:37:33.457194 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:37:33.457161 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-a3dce7/sub-test-model-a3dce7-kserve-78f7fc75f-pkntl" Nov 03 02:37:33.832470 ip-10-0-136-16 kubenswrapper[5437]: W1103 02:37:33.832407 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2cac18c4_e538_4c4b_85cf_de232f64de18.slice/crio-f2dca03c88879dad366f764e202b42cee47c8b7987b4706561589f831c74f41e WatchSource:0}: Error finding container f2dca03c88879dad366f764e202b42cee47c8b7987b4706561589f831c74f41e: Status 404 returned error can't find the container with id f2dca03c88879dad366f764e202b42cee47c8b7987b4706561589f831c74f41e Nov 03 02:37:33.835939 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:37:33.835916 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-a3dce7/sub-test-model-a3dce7-kserve-78f7fc75f-pkntl"] Nov 03 02:37:33.842211 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:37:33.842185 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-a3dce7/sub-test-model-a3dce7-kserve-78f7fc75f-pkntl" event={"ID":"2cac18c4-e538-4c4b-85cf-de232f64de18","Type":"ContainerStarted","Data":"f2dca03c88879dad366f764e202b42cee47c8b7987b4706561589f831c74f41e"} Nov 03 02:37:34.849891 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:37:34.849848 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-a3dce7/sub-test-model-a3dce7-kserve-78f7fc75f-pkntl" event={"ID":"2cac18c4-e538-4c4b-85cf-de232f64de18","Type":"ContainerStarted","Data":"dde6a8d919e274d95a2526310d31569aee9e706cfc4f2e80fa62b0b399a7ed16"} Nov 03 02:37:34.973838 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:37:34.973777 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w0-a3dce7/sub-test-model-a3dce7-kserve-78f7fc75f-pkntl" podStartSLOduration=1.973761197 podStartE2EDuration="1.973761197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:37:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:37:34.971804034 +0000 UTC m=+4737.001286328" watchObservedRunningTime="2026-11-03 02:37:34.973761197 +0000 UTC m=+4737.003243523" Nov 03 02:37:35.458090 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:37:35.458059 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w0-a3dce7/sub-test-model-a3dce7-kserve-78f7fc75f-pkntl" Nov 03 02:37:35.843913 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:37:35.843868 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-eec2ff70-55d4b96df5-zzjsd" podUID="295d5843-32eb-47f1-b942-a71a98cb2ca7" containerName="payload-processing" containerID="cri-o://5e9aab7e8457cf6bd870e5da344657638c47c1d965abbc8ce0778b70c89bbf09" gracePeriod=25 Nov 03 02:37:35.978832 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:37:35.978785 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:37:36.124852 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:37:36.124823 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-eec2ff70-55d4b96df5-zzjsd" Nov 03 02:37:36.263593 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:37:36.263562 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/295d5843-32eb-47f1-b942-a71a98cb2ca7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/295d5843-32eb-47f1-b942-a71a98cb2ca7-plugins-config-volume\") pod \"295d5843-32eb-47f1-b942-a71a98cb2ca7\" (UID: \"295d5843-32eb-47f1-b942-a71a98cb2ca7\") " Nov 03 02:37:36.263777 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:37:36.263616 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/295d5843-32eb-47f1-b942-a71a98cb2ca7-kube-api-access-dczhr\" (UniqueName: \"kubernetes.io/projected/295d5843-32eb-47f1-b942-a71a98cb2ca7-kube-api-access-dczhr\") pod \"295d5843-32eb-47f1-b942-a71a98cb2ca7\" (UID: \"295d5843-32eb-47f1-b942-a71a98cb2ca7\") " Nov 03 02:37:36.263937 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:37:36.263913 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/295d5843-32eb-47f1-b942-a71a98cb2ca7-plugins-config-volume" pod "295d5843-32eb-47f1-b942-a71a98cb2ca7" (UID: "295d5843-32eb-47f1-b942-a71a98cb2ca7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:37:36.265956 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:37:36.265925 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/295d5843-32eb-47f1-b942-a71a98cb2ca7-kube-api-access-dczhr" pod "295d5843-32eb-47f1-b942-a71a98cb2ca7" (UID: "295d5843-32eb-47f1-b942-a71a98cb2ca7"). InnerVolumeSpecName "kube-api-access-dczhr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:37:36.365160 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:37:36.365080 5437 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/295d5843-32eb-47f1-b942-a71a98cb2ca7-plugins-config-volume\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:37:36.365160 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:37:36.365110 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dczhr\" (UniqueName: \"kubernetes.io/projected/295d5843-32eb-47f1-b942-a71a98cb2ca7-kube-api-access-dczhr\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:37:36.865169 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:37:36.865134 5437 generic.go:356] "Generic (PLEG): container finished" podID="295d5843-32eb-47f1-b942-a71a98cb2ca7" containerID="5e9aab7e8457cf6bd870e5da344657638c47c1d965abbc8ce0778b70c89bbf09" exitCode=0 Nov 03 02:37:36.865423 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:37:36.865186 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-eec2ff70-55d4b96df5-zzjsd" Nov 03 02:37:36.865423 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:37:36.865222 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-eec2ff70-55d4b96df5-zzjsd" event={"ID":"295d5843-32eb-47f1-b942-a71a98cb2ca7","Type":"ContainerDied","Data":"5e9aab7e8457cf6bd870e5da344657638c47c1d965abbc8ce0778b70c89bbf09"} Nov 03 02:37:36.865423 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:37:36.865270 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-eec2ff70-55d4b96df5-zzjsd" event={"ID":"295d5843-32eb-47f1-b942-a71a98cb2ca7","Type":"ContainerDied","Data":"54bc17b18480f71b11ea278e85372c9fc7a2dccd558990175e01046191600b67"} Nov 03 02:37:36.865423 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:37:36.865298 5437 scope.go:122] "RemoveContainer" containerID="5e9aab7e8457cf6bd870e5da344657638c47c1d965abbc8ce0778b70c89bbf09" Nov 03 02:37:36.876329 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:37:36.876310 5437 scope.go:122] "RemoveContainer" containerID="5e9aab7e8457cf6bd870e5da344657638c47c1d965abbc8ce0778b70c89bbf09" Nov 03 02:37:36.876632 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:37:36.876609 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e9aab7e8457cf6bd870e5da344657638c47c1d965abbc8ce0778b70c89bbf09\": container with ID starting with 5e9aab7e8457cf6bd870e5da344657638c47c1d965abbc8ce0778b70c89bbf09 not found: ID does not exist" containerID="5e9aab7e8457cf6bd870e5da344657638c47c1d965abbc8ce0778b70c89bbf09" Nov 03 02:37:36.876744 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:37:36.876639 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e9aab7e8457cf6bd870e5da344657638c47c1d965abbc8ce0778b70c89bbf09"} err="failed to get container status \"5e9aab7e8457cf6bd870e5da344657638c47c1d965abbc8ce0778b70c89bbf09\": rpc error: code = NotFound desc = could not find container \"5e9aab7e8457cf6bd870e5da344657638c47c1d965abbc8ce0778b70c89bbf09\": container with ID starting with 5e9aab7e8457cf6bd870e5da344657638c47c1d965abbc8ce0778b70c89bbf09 not found: ID does not exist" Nov 03 02:37:37.035438 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:37:37.035404 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-eec2ff70-55d4b96df5-zzjsd"] Nov 03 02:37:37.088463 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:37:37.088431 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-eec2ff70-55d4b96df5-zzjsd"] Nov 03 02:37:38.090767 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:37:38.090736 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="295d5843-32eb-47f1-b942-a71a98cb2ca7" path="/var/lib/kubelet/pods/295d5843-32eb-47f1-b942-a71a98cb2ca7/volumes" Nov 03 02:37:40.765023 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:37:40.764980 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:37:43.116577 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:37:43.116540 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:37:43.457554 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:37:43.457456 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w0-a3dce7/sub-test-model-a3dce7-kserve-78f7fc75f-pkntl" Nov 03 02:37:43.470064 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:37:43.470034 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w0-a3dce7/sub-test-model-a3dce7-kserve-78f7fc75f-pkntl" Nov 03 02:37:43.925206 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:37:43.925173 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w0-a3dce7/sub-test-model-a3dce7-kserve-78f7fc75f-pkntl" Nov 03 02:37:44.034855 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:37:44.034818 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 02:37:45.659847 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:37:45.659804 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:37:50.374246 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:37:50.374211 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:37:55.088965 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:37:55.088923 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:37:56.343965 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:37:56.343927 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-rrtbq"] Nov 03 02:37:56.344481 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:37:56.344141 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-rrtbq" podUID="250a28b0-9884-43a0-9991-61fb8e3a5be4" containerName="istio-proxy" containerID="cri-o://e0b7a6b45540fef8d71259fad1763d12270698758371173497f6f422097c8334" gracePeriod=30 Nov 03 02:37:59.360808 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:37:59.360704 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:37:59.674922 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:37:59.674830 5437 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-rrtbq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 02:37:59.674922 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:37:59.674894 5437 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-rrtbq" podUID="250a28b0-9884-43a0-9991-61fb8e3a5be4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 02:38:00.974933 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:00.974874 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:38:01.609554 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:01.609526 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-rrtbq" Nov 03 02:38:01.661639 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:01.661551 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/250a28b0-9884-43a0-9991-61fb8e3a5be4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/250a28b0-9884-43a0-9991-61fb8e3a5be4-istio-envoy\") pod \"250a28b0-9884-43a0-9991-61fb8e3a5be4\" (UID: \"250a28b0-9884-43a0-9991-61fb8e3a5be4\") " Nov 03 02:38:01.661796 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:01.661662 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/250a28b0-9884-43a0-9991-61fb8e3a5be4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/250a28b0-9884-43a0-9991-61fb8e3a5be4-credential-socket\") pod \"250a28b0-9884-43a0-9991-61fb8e3a5be4\" (UID: \"250a28b0-9884-43a0-9991-61fb8e3a5be4\") " Nov 03 02:38:01.661796 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:01.661721 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/250a28b0-9884-43a0-9991-61fb8e3a5be4-kube-api-access-r9fq7\" (UniqueName: \"kubernetes.io/projected/250a28b0-9884-43a0-9991-61fb8e3a5be4-kube-api-access-r9fq7\") pod \"250a28b0-9884-43a0-9991-61fb8e3a5be4\" (UID: \"250a28b0-9884-43a0-9991-61fb8e3a5be4\") " Nov 03 02:38:01.661875 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:01.661805 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/250a28b0-9884-43a0-9991-61fb8e3a5be4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/250a28b0-9884-43a0-9991-61fb8e3a5be4-workload-socket\") pod \"250a28b0-9884-43a0-9991-61fb8e3a5be4\" (UID: \"250a28b0-9884-43a0-9991-61fb8e3a5be4\") " Nov 03 02:38:01.661875 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:01.661827 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/250a28b0-9884-43a0-9991-61fb8e3a5be4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/250a28b0-9884-43a0-9991-61fb8e3a5be4-workload-certs\") pod \"250a28b0-9884-43a0-9991-61fb8e3a5be4\" (UID: \"250a28b0-9884-43a0-9991-61fb8e3a5be4\") " Nov 03 02:38:01.661875 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:01.661865 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/250a28b0-9884-43a0-9991-61fb8e3a5be4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/250a28b0-9884-43a0-9991-61fb8e3a5be4-istio-podinfo\") pod \"250a28b0-9884-43a0-9991-61fb8e3a5be4\" (UID: \"250a28b0-9884-43a0-9991-61fb8e3a5be4\") " Nov 03 02:38:01.662059 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:01.661910 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/250a28b0-9884-43a0-9991-61fb8e3a5be4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/250a28b0-9884-43a0-9991-61fb8e3a5be4-istiod-ca-cert\") pod \"250a28b0-9884-43a0-9991-61fb8e3a5be4\" (UID: \"250a28b0-9884-43a0-9991-61fb8e3a5be4\") " Nov 03 02:38:01.662059 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:01.661947 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/250a28b0-9884-43a0-9991-61fb8e3a5be4-istio-token\" (UniqueName: \"kubernetes.io/projected/250a28b0-9884-43a0-9991-61fb8e3a5be4-istio-token\") pod \"250a28b0-9884-43a0-9991-61fb8e3a5be4\" (UID: \"250a28b0-9884-43a0-9991-61fb8e3a5be4\") " Nov 03 02:38:01.662059 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:01.661987 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/250a28b0-9884-43a0-9991-61fb8e3a5be4-credential-socket" pod "250a28b0-9884-43a0-9991-61fb8e3a5be4" (UID: "250a28b0-9884-43a0-9991-61fb8e3a5be4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:38:01.662059 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:01.662007 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/250a28b0-9884-43a0-9991-61fb8e3a5be4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/250a28b0-9884-43a0-9991-61fb8e3a5be4-istio-data\") pod \"250a28b0-9884-43a0-9991-61fb8e3a5be4\" (UID: \"250a28b0-9884-43a0-9991-61fb8e3a5be4\") " Nov 03 02:38:01.662278 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:01.662092 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/250a28b0-9884-43a0-9991-61fb8e3a5be4-workload-socket" pod "250a28b0-9884-43a0-9991-61fb8e3a5be4" (UID: "250a28b0-9884-43a0-9991-61fb8e3a5be4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:38:01.662278 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:01.662185 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/250a28b0-9884-43a0-9991-61fb8e3a5be4-workload-certs" pod "250a28b0-9884-43a0-9991-61fb8e3a5be4" (UID: "250a28b0-9884-43a0-9991-61fb8e3a5be4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:38:01.662278 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:01.662220 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/250a28b0-9884-43a0-9991-61fb8e3a5be4-istio-data" pod "250a28b0-9884-43a0-9991-61fb8e3a5be4" (UID: "250a28b0-9884-43a0-9991-61fb8e3a5be4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:38:01.662521 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:01.662487 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/250a28b0-9884-43a0-9991-61fb8e3a5be4-istiod-ca-cert" pod "250a28b0-9884-43a0-9991-61fb8e3a5be4" (UID: "250a28b0-9884-43a0-9991-61fb8e3a5be4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:38:01.662745 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:01.662684 5437 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/250a28b0-9884-43a0-9991-61fb8e3a5be4-workload-socket\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:38:01.662745 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:01.662712 5437 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/250a28b0-9884-43a0-9991-61fb8e3a5be4-workload-certs\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:38:01.662745 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:01.662728 5437 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/250a28b0-9884-43a0-9991-61fb8e3a5be4-istiod-ca-cert\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:38:01.662745 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:01.662741 5437 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/250a28b0-9884-43a0-9991-61fb8e3a5be4-istio-data\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:38:01.663134 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:01.662757 5437 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/250a28b0-9884-43a0-9991-61fb8e3a5be4-credential-socket\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:38:01.664233 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:01.664210 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/250a28b0-9884-43a0-9991-61fb8e3a5be4-istio-podinfo" pod "250a28b0-9884-43a0-9991-61fb8e3a5be4" (UID: "250a28b0-9884-43a0-9991-61fb8e3a5be4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:38:01.664496 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:01.664478 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/250a28b0-9884-43a0-9991-61fb8e3a5be4-kube-api-access-r9fq7" pod "250a28b0-9884-43a0-9991-61fb8e3a5be4" (UID: "250a28b0-9884-43a0-9991-61fb8e3a5be4"). InnerVolumeSpecName "kube-api-access-r9fq7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:38:01.664575 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:01.664487 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/250a28b0-9884-43a0-9991-61fb8e3a5be4-istio-envoy" pod "250a28b0-9884-43a0-9991-61fb8e3a5be4" (UID: "250a28b0-9884-43a0-9991-61fb8e3a5be4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:38:01.664575 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:01.664546 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/250a28b0-9884-43a0-9991-61fb8e3a5be4-istio-token" pod "250a28b0-9884-43a0-9991-61fb8e3a5be4" (UID: "250a28b0-9884-43a0-9991-61fb8e3a5be4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:38:01.764256 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:01.764219 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9fq7\" (UniqueName: \"kubernetes.io/projected/250a28b0-9884-43a0-9991-61fb8e3a5be4-kube-api-access-r9fq7\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:38:01.764256 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:01.764249 5437 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/250a28b0-9884-43a0-9991-61fb8e3a5be4-istio-podinfo\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:38:01.764256 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:01.764259 5437 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/250a28b0-9884-43a0-9991-61fb8e3a5be4-istio-token\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:38:01.764553 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:01.764268 5437 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/250a28b0-9884-43a0-9991-61fb8e3a5be4-istio-envoy\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:38:02.024596 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:02.024507 5437 generic.go:356] "Generic (PLEG): container finished" podID="250a28b0-9884-43a0-9991-61fb8e3a5be4" containerID="e0b7a6b45540fef8d71259fad1763d12270698758371173497f6f422097c8334" exitCode=0 Nov 03 02:38:02.024596 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:02.024558 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-rrtbq" Nov 03 02:38:02.025039 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:02.024597 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-rrtbq" event={"ID":"250a28b0-9884-43a0-9991-61fb8e3a5be4","Type":"ContainerDied","Data":"e0b7a6b45540fef8d71259fad1763d12270698758371173497f6f422097c8334"} Nov 03 02:38:02.025039 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:02.024656 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-rrtbq" event={"ID":"250a28b0-9884-43a0-9991-61fb8e3a5be4","Type":"ContainerDied","Data":"d47089d78a80c4a091785d560e6d0cbe68b71c86139f3e4bb1f2e41087936e1a"} Nov 03 02:38:02.025039 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:02.024681 5437 scope.go:122] "RemoveContainer" containerID="e0b7a6b45540fef8d71259fad1763d12270698758371173497f6f422097c8334" Nov 03 02:38:02.035860 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:02.035841 5437 scope.go:122] "RemoveContainer" containerID="e0b7a6b45540fef8d71259fad1763d12270698758371173497f6f422097c8334" Nov 03 02:38:02.036136 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:38:02.036114 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0b7a6b45540fef8d71259fad1763d12270698758371173497f6f422097c8334\": container with ID starting with e0b7a6b45540fef8d71259fad1763d12270698758371173497f6f422097c8334 not found: ID does not exist" containerID="e0b7a6b45540fef8d71259fad1763d12270698758371173497f6f422097c8334" Nov 03 02:38:02.036190 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:02.036144 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0b7a6b45540fef8d71259fad1763d12270698758371173497f6f422097c8334"} err="failed to get container status \"e0b7a6b45540fef8d71259fad1763d12270698758371173497f6f422097c8334\": rpc error: code = NotFound desc = could not find container \"e0b7a6b45540fef8d71259fad1763d12270698758371173497f6f422097c8334\": container with ID starting with e0b7a6b45540fef8d71259fad1763d12270698758371173497f6f422097c8334 not found: ID does not exist" Nov 03 02:38:02.198817 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:02.198776 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-rrtbq"] Nov 03 02:38:02.250157 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:02.250124 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-rrtbq"] Nov 03 02:38:02.919030 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:02.918987 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:38:04.089234 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:04.089195 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="250a28b0-9884-43a0-9991-61fb8e3a5be4" path="/var/lib/kubelet/pods/250a28b0-9884-43a0-9991-61fb8e3a5be4/volumes" Nov 03 02:38:05.771680 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:05.771631 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:38:07.031206 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:07.031164 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:38:07.252721 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:07.252675 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:38:07.861503 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:07.861426 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:38:07.917313 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:07.917280 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-c34879/rate-test-model-c34879-kserve-5c9b7d989-bbjkc"] Nov 03 02:38:08.068145 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:08.068110 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:38:08.557235 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:08.557194 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:38:09.242538 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:09.242502 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:38:10.058673 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:10.058635 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:38:10.779991 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:10.779959 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:38:14.036274 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:38:14.036229 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 02:38:15.714831 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:15.714788 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:38:16.300806 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:16.300759 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:38:16.520072 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:16.520035 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-059b23-74cdd47845-8jxpp"] Nov 03 02:38:16.928203 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:16.928108 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:38:17.134763 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:17.134705 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:38:20.727872 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:20.727834 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:38:21.520755 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:21.520701 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-059b23-74cdd47845-8jxpp" podUID="f622668a-42fc-4471-9b7f-1166924c2769" containerName="payload-processing" containerID="cri-o://45c70dbd38645b993d840ab9a97d78af99dd76b4971696776eccab878efba977" gracePeriod=25 Nov 03 02:38:22.321422 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:22.321392 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:38:22.938077 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:22.938031 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-c34879/rate-test-model-c34879-kserve-5c9b7d989-bbjkc" podUID="ff3f6c0f-5c72-4897-ac51-f9cef66c9f95" containerName="main" containerID="cri-o://f234defaf896033a249ce38922ebba5775738c06eaa362a66895ecf5a88bdbfd" gracePeriod=45 Nov 03 02:38:22.991917 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:22.991876 5437 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-c34879/rate-test-model-c34879-kserve-5c9b7d989-bbjkc" podUID="ff3f6c0f-5c72-4897-ac51-f9cef66c9f95" containerName="main" probeResult="failure" output="Get \"https://10.134.0.183:8000/ready\": dial tcp 10.134.0.183:8000: connect: connection refused" Nov 03 02:38:23.157428 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:23.157386 5437 generic.go:356] "Generic (PLEG): container finished" podID="ff3f6c0f-5c72-4897-ac51-f9cef66c9f95" containerID="f234defaf896033a249ce38922ebba5775738c06eaa362a66895ecf5a88bdbfd" exitCode=0 Nov 03 02:38:23.157601 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:23.157471 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-c34879/rate-test-model-c34879-kserve-5c9b7d989-bbjkc" event={"ID":"ff3f6c0f-5c72-4897-ac51-f9cef66c9f95","Type":"ContainerDied","Data":"f234defaf896033a249ce38922ebba5775738c06eaa362a66895ecf5a88bdbfd"} Nov 03 02:38:23.209917 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:23.209889 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-c34879/rate-test-model-c34879-kserve-5c9b7d989-bbjkc" Nov 03 02:38:23.321390 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:23.321332 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff3f6c0f-5c72-4897-ac51-f9cef66c9f95-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff3f6c0f-5c72-4897-ac51-f9cef66c9f95-model-cache\") pod \"ff3f6c0f-5c72-4897-ac51-f9cef66c9f95\" (UID: \"ff3f6c0f-5c72-4897-ac51-f9cef66c9f95\") " Nov 03 02:38:23.321559 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:23.321458 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff3f6c0f-5c72-4897-ac51-f9cef66c9f95-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff3f6c0f-5c72-4897-ac51-f9cef66c9f95-dshm\") pod \"ff3f6c0f-5c72-4897-ac51-f9cef66c9f95\" (UID: \"ff3f6c0f-5c72-4897-ac51-f9cef66c9f95\") " Nov 03 02:38:23.321559 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:23.321520 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff3f6c0f-5c72-4897-ac51-f9cef66c9f95-home\" (UniqueName: \"kubernetes.io/empty-dir/ff3f6c0f-5c72-4897-ac51-f9cef66c9f95-home\") pod \"ff3f6c0f-5c72-4897-ac51-f9cef66c9f95\" (UID: \"ff3f6c0f-5c72-4897-ac51-f9cef66c9f95\") " Nov 03 02:38:23.322019 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:23.321590 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff3f6c0f-5c72-4897-ac51-f9cef66c9f95-tls-certs\" (UniqueName: \"kubernetes.io/secret/ff3f6c0f-5c72-4897-ac51-f9cef66c9f95-tls-certs\") pod \"ff3f6c0f-5c72-4897-ac51-f9cef66c9f95\" (UID: \"ff3f6c0f-5c72-4897-ac51-f9cef66c9f95\") " Nov 03 02:38:23.322019 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:23.321642 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff3f6c0f-5c72-4897-ac51-f9cef66c9f95-model-cache" pod "ff3f6c0f-5c72-4897-ac51-f9cef66c9f95" (UID: "ff3f6c0f-5c72-4897-ac51-f9cef66c9f95"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:38:23.322019 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:23.321655 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff3f6c0f-5c72-4897-ac51-f9cef66c9f95-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff3f6c0f-5c72-4897-ac51-f9cef66c9f95-tmp-dir\") pod \"ff3f6c0f-5c72-4897-ac51-f9cef66c9f95\" (UID: \"ff3f6c0f-5c72-4897-ac51-f9cef66c9f95\") " Nov 03 02:38:23.322019 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:23.321807 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff3f6c0f-5c72-4897-ac51-f9cef66c9f95-home" pod "ff3f6c0f-5c72-4897-ac51-f9cef66c9f95" (UID: "ff3f6c0f-5c72-4897-ac51-f9cef66c9f95"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:38:23.322019 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:23.321826 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff3f6c0f-5c72-4897-ac51-f9cef66c9f95-kube-api-access-9x7ns\" (UniqueName: \"kubernetes.io/projected/ff3f6c0f-5c72-4897-ac51-f9cef66c9f95-kube-api-access-9x7ns\") pod \"ff3f6c0f-5c72-4897-ac51-f9cef66c9f95\" (UID: \"ff3f6c0f-5c72-4897-ac51-f9cef66c9f95\") " Nov 03 02:38:23.322019 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:23.321841 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff3f6c0f-5c72-4897-ac51-f9cef66c9f95-tmp-dir" pod "ff3f6c0f-5c72-4897-ac51-f9cef66c9f95" (UID: "ff3f6c0f-5c72-4897-ac51-f9cef66c9f95"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:38:23.322382 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:23.322338 5437 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff3f6c0f-5c72-4897-ac51-f9cef66c9f95-model-cache\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:38:23.322445 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:23.322426 5437 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff3f6c0f-5c72-4897-ac51-f9cef66c9f95-home\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:38:23.322482 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:23.322469 5437 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff3f6c0f-5c72-4897-ac51-f9cef66c9f95-tmp-dir\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:38:23.323836 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:23.323804 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff3f6c0f-5c72-4897-ac51-f9cef66c9f95-kube-api-access-9x7ns" pod "ff3f6c0f-5c72-4897-ac51-f9cef66c9f95" (UID: "ff3f6c0f-5c72-4897-ac51-f9cef66c9f95"). InnerVolumeSpecName "kube-api-access-9x7ns". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:38:23.323939 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:23.323836 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff3f6c0f-5c72-4897-ac51-f9cef66c9f95-dshm" pod "ff3f6c0f-5c72-4897-ac51-f9cef66c9f95" (UID: "ff3f6c0f-5c72-4897-ac51-f9cef66c9f95"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:38:23.323939 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:23.323864 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff3f6c0f-5c72-4897-ac51-f9cef66c9f95-tls-certs" pod "ff3f6c0f-5c72-4897-ac51-f9cef66c9f95" (UID: "ff3f6c0f-5c72-4897-ac51-f9cef66c9f95"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:38:23.423316 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:23.423282 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9x7ns\" (UniqueName: \"kubernetes.io/projected/ff3f6c0f-5c72-4897-ac51-f9cef66c9f95-kube-api-access-9x7ns\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:38:23.423316 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:23.423313 5437 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff3f6c0f-5c72-4897-ac51-f9cef66c9f95-dshm\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:38:23.423316 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:23.423323 5437 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff3f6c0f-5c72-4897-ac51-f9cef66c9f95-tls-certs\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:38:24.164960 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:24.164926 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-c34879/rate-test-model-c34879-kserve-5c9b7d989-bbjkc" Nov 03 02:38:24.165159 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:24.164960 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-c34879/rate-test-model-c34879-kserve-5c9b7d989-bbjkc" event={"ID":"ff3f6c0f-5c72-4897-ac51-f9cef66c9f95","Type":"ContainerDied","Data":"2018fbce9efbfdeec983d29e69f898d1ce3072ef085ddeb07b7f2fff5c25d587"} Nov 03 02:38:24.165159 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:24.165012 5437 scope.go:122] "RemoveContainer" containerID="f234defaf896033a249ce38922ebba5775738c06eaa362a66895ecf5a88bdbfd" Nov 03 02:38:24.335614 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:24.335575 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-c34879/rate-test-model-c34879-kserve-5c9b7d989-bbjkc"] Nov 03 02:38:24.386673 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:24.386580 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-c34879/rate-test-model-c34879-kserve-5c9b7d989-bbjkc"] Nov 03 02:38:25.502065 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:25.501982 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:38:26.089499 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:26.089463 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff3f6c0f-5c72-4897-ac51-f9cef66c9f95" path="/var/lib/kubelet/pods/ff3f6c0f-5c72-4897-ac51-f9cef66c9f95/volumes" Nov 03 02:38:27.616437 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:27.616350 5437 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-059b23-74cdd47845-8jxpp" podUID="f622668a-42fc-4471-9b7f-1166924c2769" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 02:38:30.295957 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:30.295914 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:38:35.034243 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:35.034203 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:38:37.058691 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:37.058658 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-d9576a/rate-test-model-d9576a-kserve-779b5b8fc-lzrmc"] Nov 03 02:38:37.186629 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:37.186593 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:38:37.610055 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:37.610023 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:38:37.615814 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:37.615777 5437 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-059b23-74cdd47845-8jxpp" podUID="f622668a-42fc-4471-9b7f-1166924c2769" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 02:38:39.872161 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:39.872063 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:38:41.790447 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:41.790348 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-059b23-74cdd47845-8jxpp" Nov 03 02:38:41.933576 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:41.933536 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f622668a-42fc-4471-9b7f-1166924c2769-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f622668a-42fc-4471-9b7f-1166924c2769-plugins-config-volume\") pod \"f622668a-42fc-4471-9b7f-1166924c2769\" (UID: \"f622668a-42fc-4471-9b7f-1166924c2769\") " Nov 03 02:38:41.933783 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:41.933764 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f622668a-42fc-4471-9b7f-1166924c2769-kube-api-access-nj6bg\" (UniqueName: \"kubernetes.io/projected/f622668a-42fc-4471-9b7f-1166924c2769-kube-api-access-nj6bg\") pod \"f622668a-42fc-4471-9b7f-1166924c2769\" (UID: \"f622668a-42fc-4471-9b7f-1166924c2769\") " Nov 03 02:38:41.934026 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:41.934000 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f622668a-42fc-4471-9b7f-1166924c2769-plugins-config-volume" pod "f622668a-42fc-4471-9b7f-1166924c2769" (UID: "f622668a-42fc-4471-9b7f-1166924c2769"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:38:41.934305 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:41.934280 5437 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f622668a-42fc-4471-9b7f-1166924c2769-plugins-config-volume\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:38:41.936056 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:41.936024 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f622668a-42fc-4471-9b7f-1166924c2769-kube-api-access-nj6bg" pod "f622668a-42fc-4471-9b7f-1166924c2769" (UID: "f622668a-42fc-4471-9b7f-1166924c2769"). InnerVolumeSpecName "kube-api-access-nj6bg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:38:42.035962 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:42.035921 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nj6bg\" (UniqueName: \"kubernetes.io/projected/f622668a-42fc-4471-9b7f-1166924c2769-kube-api-access-nj6bg\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:38:42.271874 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:42.271839 5437 generic.go:356] "Generic (PLEG): container finished" podID="f622668a-42fc-4471-9b7f-1166924c2769" containerID="45c70dbd38645b993d840ab9a97d78af99dd76b4971696776eccab878efba977" exitCode=0 Nov 03 02:38:42.272056 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:42.271889 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-059b23-74cdd47845-8jxpp" Nov 03 02:38:42.272056 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:42.271927 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-059b23-74cdd47845-8jxpp" event={"ID":"f622668a-42fc-4471-9b7f-1166924c2769","Type":"ContainerDied","Data":"45c70dbd38645b993d840ab9a97d78af99dd76b4971696776eccab878efba977"} Nov 03 02:38:42.272056 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:42.271980 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-059b23-74cdd47845-8jxpp" event={"ID":"f622668a-42fc-4471-9b7f-1166924c2769","Type":"ContainerDied","Data":"5b10727791f122f20b7b980abb5f2048f52166d128056c1b5103ffaabede29e9"} Nov 03 02:38:42.272056 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:42.272001 5437 scope.go:122] "RemoveContainer" containerID="45c70dbd38645b993d840ab9a97d78af99dd76b4971696776eccab878efba977" Nov 03 02:38:42.282101 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:42.282079 5437 scope.go:122] "RemoveContainer" containerID="45c70dbd38645b993d840ab9a97d78af99dd76b4971696776eccab878efba977" Nov 03 02:38:42.282388 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:38:42.282353 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45c70dbd38645b993d840ab9a97d78af99dd76b4971696776eccab878efba977\": container with ID starting with 45c70dbd38645b993d840ab9a97d78af99dd76b4971696776eccab878efba977 not found: ID does not exist" containerID="45c70dbd38645b993d840ab9a97d78af99dd76b4971696776eccab878efba977" Nov 03 02:38:42.282445 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:42.282397 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45c70dbd38645b993d840ab9a97d78af99dd76b4971696776eccab878efba977"} err="failed to get container status \"45c70dbd38645b993d840ab9a97d78af99dd76b4971696776eccab878efba977\": rpc error: code = NotFound desc = could not find container \"45c70dbd38645b993d840ab9a97d78af99dd76b4971696776eccab878efba977\": container with ID starting with 45c70dbd38645b993d840ab9a97d78af99dd76b4971696776eccab878efba977 not found: ID does not exist" Nov 03 02:38:42.442112 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:42.442023 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-059b23-74cdd47845-8jxpp"] Nov 03 02:38:42.445260 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:42.445215 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-059b23-74cdd47845-8jxpp"] Nov 03 02:38:42.637545 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:42.637513 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-c34879-8655cb9dd9-gl2ss"] Nov 03 02:38:42.739211 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:42.739123 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c34879-6ff9ff78b7-sk2j2"] Nov 03 02:38:43.057238 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:43.057198 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:38:43.271797 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:43.271759 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:38:44.036233 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:38:44.036187 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 02:38:44.089359 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:44.089325 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f622668a-42fc-4471-9b7f-1166924c2769" path="/var/lib/kubelet/pods/f622668a-42fc-4471-9b7f-1166924c2769/volumes" Nov 03 02:38:44.197632 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:44.197596 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:38:44.939860 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:44.939825 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:38:47.638667 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:47.638608 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-c34879-8655cb9dd9-gl2ss" podUID="4cd50b12-718f-4a10-a4f5-16a17fc90342" containerName="payload-processing" containerID="cri-o://d2605c4ed68964e7a345e5ee3478369ebfd139ad35f06157b67f36eefad350ea" gracePeriod=25 Nov 03 02:38:47.739462 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:47.739414 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c34879-6ff9ff78b7-sk2j2" podUID="c3d40c05-1fca-4cad-873c-cc68bc420627" containerName="payload-pre-processing" containerID="cri-o://23b797ac5a126cdd03d1edd848a0f3b7983cce94e062135fd227ebac4ca7ef8e" gracePeriod=25 Nov 03 02:38:47.770822 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:47.770778 5437 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-c34879-8655cb9dd9-gl2ss" podUID="4cd50b12-718f-4a10-a4f5-16a17fc90342" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 02:38:47.770822 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:47.770810 5437 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c34879-6ff9ff78b7-sk2j2" podUID="c3d40c05-1fca-4cad-873c-cc68bc420627" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 02:38:49.767456 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:49.767417 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:38:50.019906 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:50.019811 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:38:52.079344 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:52.079294 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-d9576a/rate-test-model-d9576a-kserve-779b5b8fc-lzrmc" podUID="e1907086-a8a1-40a5-9a85-8b9d9ea21e40" containerName="main" containerID="cri-o://c55116364beede002ac238a51fed534cb9dbcb619c934c76ea6ec13cab0a8b66" gracePeriod=45 Nov 03 02:38:52.334481 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:52.334450 5437 generic.go:356] "Generic (PLEG): container finished" podID="e1907086-a8a1-40a5-9a85-8b9d9ea21e40" containerID="c55116364beede002ac238a51fed534cb9dbcb619c934c76ea6ec13cab0a8b66" exitCode=0 Nov 03 02:38:52.334577 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:52.334507 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-d9576a/rate-test-model-d9576a-kserve-779b5b8fc-lzrmc" event={"ID":"e1907086-a8a1-40a5-9a85-8b9d9ea21e40","Type":"ContainerDied","Data":"c55116364beede002ac238a51fed534cb9dbcb619c934c76ea6ec13cab0a8b66"} Nov 03 02:38:52.353429 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:52.353398 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-d9576a/rate-test-model-d9576a-kserve-779b5b8fc-lzrmc" Nov 03 02:38:52.450412 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:52.450373 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1907086-a8a1-40a5-9a85-8b9d9ea21e40-kube-api-access-lr9dw\" (UniqueName: \"kubernetes.io/projected/e1907086-a8a1-40a5-9a85-8b9d9ea21e40-kube-api-access-lr9dw\") pod \"e1907086-a8a1-40a5-9a85-8b9d9ea21e40\" (UID: \"e1907086-a8a1-40a5-9a85-8b9d9ea21e40\") " Nov 03 02:38:52.450574 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:52.450526 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1907086-a8a1-40a5-9a85-8b9d9ea21e40-home\" (UniqueName: \"kubernetes.io/empty-dir/e1907086-a8a1-40a5-9a85-8b9d9ea21e40-home\") pod \"e1907086-a8a1-40a5-9a85-8b9d9ea21e40\" (UID: \"e1907086-a8a1-40a5-9a85-8b9d9ea21e40\") " Nov 03 02:38:52.450637 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:52.450579 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1907086-a8a1-40a5-9a85-8b9d9ea21e40-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1907086-a8a1-40a5-9a85-8b9d9ea21e40-model-cache\") pod \"e1907086-a8a1-40a5-9a85-8b9d9ea21e40\" (UID: \"e1907086-a8a1-40a5-9a85-8b9d9ea21e40\") " Nov 03 02:38:52.450785 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:52.450769 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1907086-a8a1-40a5-9a85-8b9d9ea21e40-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1907086-a8a1-40a5-9a85-8b9d9ea21e40-tmp-dir\") pod \"e1907086-a8a1-40a5-9a85-8b9d9ea21e40\" (UID: \"e1907086-a8a1-40a5-9a85-8b9d9ea21e40\") " Nov 03 02:38:52.450879 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:52.450779 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1907086-a8a1-40a5-9a85-8b9d9ea21e40-home" pod "e1907086-a8a1-40a5-9a85-8b9d9ea21e40" (UID: "e1907086-a8a1-40a5-9a85-8b9d9ea21e40"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:38:52.450879 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:52.450845 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1907086-a8a1-40a5-9a85-8b9d9ea21e40-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1907086-a8a1-40a5-9a85-8b9d9ea21e40-dshm\") pod \"e1907086-a8a1-40a5-9a85-8b9d9ea21e40\" (UID: \"e1907086-a8a1-40a5-9a85-8b9d9ea21e40\") " Nov 03 02:38:52.450988 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:52.450885 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e1907086-a8a1-40a5-9a85-8b9d9ea21e40-tls-certs\" (UniqueName: \"kubernetes.io/secret/e1907086-a8a1-40a5-9a85-8b9d9ea21e40-tls-certs\") pod \"e1907086-a8a1-40a5-9a85-8b9d9ea21e40\" (UID: \"e1907086-a8a1-40a5-9a85-8b9d9ea21e40\") " Nov 03 02:38:52.450988 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:52.450892 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1907086-a8a1-40a5-9a85-8b9d9ea21e40-model-cache" pod "e1907086-a8a1-40a5-9a85-8b9d9ea21e40" (UID: "e1907086-a8a1-40a5-9a85-8b9d9ea21e40"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:38:52.451097 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:52.450988 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1907086-a8a1-40a5-9a85-8b9d9ea21e40-tmp-dir" pod "e1907086-a8a1-40a5-9a85-8b9d9ea21e40" (UID: "e1907086-a8a1-40a5-9a85-8b9d9ea21e40"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:38:52.451328 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:52.451305 5437 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1907086-a8a1-40a5-9a85-8b9d9ea21e40-tmp-dir\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:38:52.451429 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:52.451331 5437 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1907086-a8a1-40a5-9a85-8b9d9ea21e40-home\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:38:52.451429 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:52.451344 5437 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1907086-a8a1-40a5-9a85-8b9d9ea21e40-model-cache\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:38:52.452920 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:52.452892 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1907086-a8a1-40a5-9a85-8b9d9ea21e40-kube-api-access-lr9dw" pod "e1907086-a8a1-40a5-9a85-8b9d9ea21e40" (UID: "e1907086-a8a1-40a5-9a85-8b9d9ea21e40"). InnerVolumeSpecName "kube-api-access-lr9dw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:38:52.453159 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:52.453135 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1907086-a8a1-40a5-9a85-8b9d9ea21e40-tls-certs" pod "e1907086-a8a1-40a5-9a85-8b9d9ea21e40" (UID: "e1907086-a8a1-40a5-9a85-8b9d9ea21e40"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:38:52.453293 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:52.453156 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1907086-a8a1-40a5-9a85-8b9d9ea21e40-dshm" pod "e1907086-a8a1-40a5-9a85-8b9d9ea21e40" (UID: "e1907086-a8a1-40a5-9a85-8b9d9ea21e40"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:38:52.552779 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:52.552743 5437 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1907086-a8a1-40a5-9a85-8b9d9ea21e40-dshm\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:38:52.552779 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:52.552773 5437 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1907086-a8a1-40a5-9a85-8b9d9ea21e40-tls-certs\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:38:52.552779 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:52.552785 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lr9dw\" (UniqueName: \"kubernetes.io/projected/e1907086-a8a1-40a5-9a85-8b9d9ea21e40-kube-api-access-lr9dw\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:38:53.342856 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:53.342817 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-d9576a/rate-test-model-d9576a-kserve-779b5b8fc-lzrmc" event={"ID":"e1907086-a8a1-40a5-9a85-8b9d9ea21e40","Type":"ContainerDied","Data":"0b67061ee7eb40f2febef7605e1fc6769ef3b7f8aa00857ec80c46c95763ca80"} Nov 03 02:38:53.343393 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:53.342880 5437 scope.go:122] "RemoveContainer" containerID="c55116364beede002ac238a51fed534cb9dbcb619c934c76ea6ec13cab0a8b66" Nov 03 02:38:53.343393 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:53.342829 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-d9576a/rate-test-model-d9576a-kserve-779b5b8fc-lzrmc" Nov 03 02:38:53.512636 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:53.512600 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-d9576a/rate-test-model-d9576a-kserve-779b5b8fc-lzrmc"] Nov 03 02:38:53.563957 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:53.563915 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-d9576a/rate-test-model-d9576a-kserve-779b5b8fc-lzrmc"] Nov 03 02:38:54.088283 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:54.088249 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1907086-a8a1-40a5-9a85-8b9d9ea21e40" path="/var/lib/kubelet/pods/e1907086-a8a1-40a5-9a85-8b9d9ea21e40/volumes" Nov 03 02:38:54.591975 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:54.591942 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:38:57.770589 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:57.770546 5437 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c34879-6ff9ff78b7-sk2j2" podUID="c3d40c05-1fca-4cad-873c-cc68bc420627" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 02:38:57.770589 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:57.770565 5437 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-c34879-8655cb9dd9-gl2ss" podUID="4cd50b12-718f-4a10-a4f5-16a17fc90342" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 02:38:59.371716 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:38:59.371637 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:39:03.258995 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:03.258957 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:39:03.459489 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:03.459446 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:39:03.720948 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:03.720910 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:39:05.223961 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:05.223928 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:39:06.018484 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:06.018440 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-98a394/rate-test-model-98a394-kserve-6d7884c4c7-gp5g4"] Nov 03 02:39:06.082574 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:06.082541 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:39:06.451037 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:06.450975 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:39:06.652169 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:06.652124 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:39:07.770881 ip-10-0-136-16 kubenswrapper[5437]: W1103 02:39:07.770854 5437 logging.go:55] [core] [Channel #16590 SubChannel #16591]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.130:50052", ServerName: "10.134.0.130:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.130:50052: connect: connection refused" Nov 03 02:39:07.771251 ip-10-0-136-16 kubenswrapper[5437]: W1103 02:39:07.770886 5437 logging.go:55] [core] [Channel #16589 SubChannel #16592]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.129:50052", ServerName: "10.134.0.129:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.129:50052: connect: connection refused" Nov 03 02:39:07.900167 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:07.900098 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-c34879-8655cb9dd9-gl2ss" Nov 03 02:39:07.988767 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:07.988736 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:39:07.995344 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:07.995320 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c34879-6ff9ff78b7-sk2j2" Nov 03 02:39:08.034837 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:08.034791 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4cd50b12-718f-4a10-a4f5-16a17fc90342-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cd50b12-718f-4a10-a4f5-16a17fc90342-plugins-config-volume\") pod \"4cd50b12-718f-4a10-a4f5-16a17fc90342\" (UID: \"4cd50b12-718f-4a10-a4f5-16a17fc90342\") " Nov 03 02:39:08.035028 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:08.035015 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cd50b12-718f-4a10-a4f5-16a17fc90342-kube-api-access-xr72b\" (UniqueName: \"kubernetes.io/projected/4cd50b12-718f-4a10-a4f5-16a17fc90342-kube-api-access-xr72b\") pod \"4cd50b12-718f-4a10-a4f5-16a17fc90342\" (UID: \"4cd50b12-718f-4a10-a4f5-16a17fc90342\") " Nov 03 02:39:08.035278 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:08.035251 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cd50b12-718f-4a10-a4f5-16a17fc90342-plugins-config-volume" pod "4cd50b12-718f-4a10-a4f5-16a17fc90342" (UID: "4cd50b12-718f-4a10-a4f5-16a17fc90342"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:39:08.035525 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:08.035501 5437 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cd50b12-718f-4a10-a4f5-16a17fc90342-plugins-config-volume\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:39:08.037319 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:08.037291 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cd50b12-718f-4a10-a4f5-16a17fc90342-kube-api-access-xr72b" pod "4cd50b12-718f-4a10-a4f5-16a17fc90342" (UID: "4cd50b12-718f-4a10-a4f5-16a17fc90342"). InnerVolumeSpecName "kube-api-access-xr72b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:39:08.136012 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:08.135974 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3d40c05-1fca-4cad-873c-cc68bc420627-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3d40c05-1fca-4cad-873c-cc68bc420627-plugins-config-volume\") pod \"c3d40c05-1fca-4cad-873c-cc68bc420627\" (UID: \"c3d40c05-1fca-4cad-873c-cc68bc420627\") " Nov 03 02:39:08.136199 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:08.136100 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3d40c05-1fca-4cad-873c-cc68bc420627-kube-api-access-p5vfz\" (UniqueName: \"kubernetes.io/projected/c3d40c05-1fca-4cad-873c-cc68bc420627-kube-api-access-p5vfz\") pod \"c3d40c05-1fca-4cad-873c-cc68bc420627\" (UID: \"c3d40c05-1fca-4cad-873c-cc68bc420627\") " Nov 03 02:39:08.136431 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:08.136399 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3d40c05-1fca-4cad-873c-cc68bc420627-plugins-config-volume" pod "c3d40c05-1fca-4cad-873c-cc68bc420627" (UID: "c3d40c05-1fca-4cad-873c-cc68bc420627"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:39:08.136867 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:08.136703 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xr72b\" (UniqueName: \"kubernetes.io/projected/4cd50b12-718f-4a10-a4f5-16a17fc90342-kube-api-access-xr72b\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:39:08.136867 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:08.136731 5437 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3d40c05-1fca-4cad-873c-cc68bc420627-plugins-config-volume\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:39:08.138321 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:08.138294 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3d40c05-1fca-4cad-873c-cc68bc420627-kube-api-access-p5vfz" pod "c3d40c05-1fca-4cad-873c-cc68bc420627" (UID: "c3d40c05-1fca-4cad-873c-cc68bc420627"). InnerVolumeSpecName "kube-api-access-p5vfz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:39:08.215300 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:08.215210 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:39:08.238545 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:08.238510 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5vfz\" (UniqueName: \"kubernetes.io/projected/c3d40c05-1fca-4cad-873c-cc68bc420627-kube-api-access-p5vfz\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:39:08.439425 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:08.439387 5437 generic.go:356] "Generic (PLEG): container finished" podID="c3d40c05-1fca-4cad-873c-cc68bc420627" containerID="23b797ac5a126cdd03d1edd848a0f3b7983cce94e062135fd227ebac4ca7ef8e" exitCode=0 Nov 03 02:39:08.439639 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:08.439437 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c34879-6ff9ff78b7-sk2j2" Nov 03 02:39:08.439639 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:08.439482 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c34879-6ff9ff78b7-sk2j2" event={"ID":"c3d40c05-1fca-4cad-873c-cc68bc420627","Type":"ContainerDied","Data":"23b797ac5a126cdd03d1edd848a0f3b7983cce94e062135fd227ebac4ca7ef8e"} Nov 03 02:39:08.439639 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:08.439553 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c34879-6ff9ff78b7-sk2j2" event={"ID":"c3d40c05-1fca-4cad-873c-cc68bc420627","Type":"ContainerDied","Data":"b2158318de51252c0caa8999e21a350b8704b1010e634dc74f1097ae502324be"} Nov 03 02:39:08.439639 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:08.439589 5437 scope.go:122] "RemoveContainer" containerID="23b797ac5a126cdd03d1edd848a0f3b7983cce94e062135fd227ebac4ca7ef8e" Nov 03 02:39:08.441135 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:08.441107 5437 generic.go:356] "Generic (PLEG): container finished" podID="4cd50b12-718f-4a10-a4f5-16a17fc90342" containerID="d2605c4ed68964e7a345e5ee3478369ebfd139ad35f06157b67f36eefad350ea" exitCode=0 Nov 03 02:39:08.441281 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:08.441149 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-c34879-8655cb9dd9-gl2ss" Nov 03 02:39:08.441281 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:08.441180 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-c34879-8655cb9dd9-gl2ss" event={"ID":"4cd50b12-718f-4a10-a4f5-16a17fc90342","Type":"ContainerDied","Data":"d2605c4ed68964e7a345e5ee3478369ebfd139ad35f06157b67f36eefad350ea"} Nov 03 02:39:08.441281 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:08.441220 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-c34879-8655cb9dd9-gl2ss" event={"ID":"4cd50b12-718f-4a10-a4f5-16a17fc90342","Type":"ContainerDied","Data":"e857ea1f406edc4a04aa17d1d40a7c1a584de3a26944cceac7cefaaf87d49f17"} Nov 03 02:39:08.450540 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:08.450515 5437 scope.go:122] "RemoveContainer" containerID="23b797ac5a126cdd03d1edd848a0f3b7983cce94e062135fd227ebac4ca7ef8e" Nov 03 02:39:08.450810 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:39:08.450789 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23b797ac5a126cdd03d1edd848a0f3b7983cce94e062135fd227ebac4ca7ef8e\": container with ID starting with 23b797ac5a126cdd03d1edd848a0f3b7983cce94e062135fd227ebac4ca7ef8e not found: ID does not exist" containerID="23b797ac5a126cdd03d1edd848a0f3b7983cce94e062135fd227ebac4ca7ef8e" Nov 03 02:39:08.450884 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:08.450819 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23b797ac5a126cdd03d1edd848a0f3b7983cce94e062135fd227ebac4ca7ef8e"} err="failed to get container status \"23b797ac5a126cdd03d1edd848a0f3b7983cce94e062135fd227ebac4ca7ef8e\": rpc error: code = NotFound desc = could not find container \"23b797ac5a126cdd03d1edd848a0f3b7983cce94e062135fd227ebac4ca7ef8e\": container with ID starting with 23b797ac5a126cdd03d1edd848a0f3b7983cce94e062135fd227ebac4ca7ef8e not found: ID does not exist" Nov 03 02:39:08.450884 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:08.450838 5437 scope.go:122] "RemoveContainer" containerID="d2605c4ed68964e7a345e5ee3478369ebfd139ad35f06157b67f36eefad350ea" Nov 03 02:39:08.460751 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:08.460730 5437 scope.go:122] "RemoveContainer" containerID="d2605c4ed68964e7a345e5ee3478369ebfd139ad35f06157b67f36eefad350ea" Nov 03 02:39:08.461073 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:39:08.461052 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2605c4ed68964e7a345e5ee3478369ebfd139ad35f06157b67f36eefad350ea\": container with ID starting with d2605c4ed68964e7a345e5ee3478369ebfd139ad35f06157b67f36eefad350ea not found: ID does not exist" containerID="d2605c4ed68964e7a345e5ee3478369ebfd139ad35f06157b67f36eefad350ea" Nov 03 02:39:08.461138 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:08.461082 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2605c4ed68964e7a345e5ee3478369ebfd139ad35f06157b67f36eefad350ea"} err="failed to get container status \"d2605c4ed68964e7a345e5ee3478369ebfd139ad35f06157b67f36eefad350ea\": rpc error: code = NotFound desc = could not find container \"d2605c4ed68964e7a345e5ee3478369ebfd139ad35f06157b67f36eefad350ea\": container with ID starting with d2605c4ed68964e7a345e5ee3478369ebfd139ad35f06157b67f36eefad350ea not found: ID does not exist" Nov 03 02:39:08.482777 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:08.482702 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:39:08.610940 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:08.610907 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c34879-6ff9ff78b7-sk2j2"] Nov 03 02:39:08.614256 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:08.614228 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c34879-6ff9ff78b7-sk2j2"] Nov 03 02:39:08.727825 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:08.727788 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-c34879-8655cb9dd9-gl2ss"] Nov 03 02:39:08.733280 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:08.733205 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-c34879-8655cb9dd9-gl2ss"] Nov 03 02:39:08.771117 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:08.771066 5437 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c34879-6ff9ff78b7-sk2j2" podUID="c3d40c05-1fca-4cad-873c-cc68bc420627" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.129:50052\" within 1s: context deadline exceeded" Nov 03 02:39:08.771513 ip-10-0-136-16 kubenswrapper[5437]: W1103 02:39:08.771129 5437 logging.go:55] [core] [Channel #16589 SubChannel #16592]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.129:50052", ServerName: "10.134.0.129:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.129:50052: operation was canceled" Nov 03 02:39:08.771513 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:08.771071 5437 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-c34879-8655cb9dd9-gl2ss" podUID="4cd50b12-718f-4a10-a4f5-16a17fc90342" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.130:50052\" within 1s: context deadline exceeded" Nov 03 02:39:08.771513 ip-10-0-136-16 kubenswrapper[5437]: W1103 02:39:08.771197 5437 logging.go:55] [core] [Channel #16590 SubChannel #16591]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.130:50052", ServerName: "10.134.0.130:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.130:50052: operation was canceled" Nov 03 02:39:09.261578 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:09.261529 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:39:10.089191 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:10.089155 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4cd50b12-718f-4a10-a4f5-16a17fc90342" path="/var/lib/kubelet/pods/4cd50b12-718f-4a10-a4f5-16a17fc90342/volumes" Nov 03 02:39:10.089672 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:10.089656 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3d40c05-1fca-4cad-873c-cc68bc420627" path="/var/lib/kubelet/pods/c3d40c05-1fca-4cad-873c-cc68bc420627/volumes" Nov 03 02:39:10.277758 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:10.277720 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:39:14.035302 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:39:14.035257 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 02:39:14.710747 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:14.710711 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:39:14.915906 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:14.915854 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:39:15.331333 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:15.331291 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:39:15.534097 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:15.534048 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:39:16.781502 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:16.781460 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:39:17.032485 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:17.032393 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:39:18.075184 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:18.075149 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:39:18.281915 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:18.281878 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:39:20.193342 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:20.193308 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:39:20.511772 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:20.511664 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:39:21.039212 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:21.039153 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-98a394/rate-test-model-98a394-kserve-6d7884c4c7-gp5g4" podUID="786969b9-72c9-4b56-91e4-97d255bbe6e4" containerName="main" containerID="cri-o://6b5fddaf6969ccbec71ef3f69e35af11ec74b017e065ac02ed99bff495aa6670" gracePeriod=45 Nov 03 02:39:21.297265 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:21.297243 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-98a394/rate-test-model-98a394-kserve-6d7884c4c7-gp5g4" Nov 03 02:39:21.394613 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:21.394575 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/786969b9-72c9-4b56-91e4-97d255bbe6e4-kube-api-access-v8gpr\" (UniqueName: \"kubernetes.io/projected/786969b9-72c9-4b56-91e4-97d255bbe6e4-kube-api-access-v8gpr\") pod \"786969b9-72c9-4b56-91e4-97d255bbe6e4\" (UID: \"786969b9-72c9-4b56-91e4-97d255bbe6e4\") " Nov 03 02:39:21.394803 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:21.394694 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/786969b9-72c9-4b56-91e4-97d255bbe6e4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/786969b9-72c9-4b56-91e4-97d255bbe6e4-dshm\") pod \"786969b9-72c9-4b56-91e4-97d255bbe6e4\" (UID: \"786969b9-72c9-4b56-91e4-97d255bbe6e4\") " Nov 03 02:39:21.394803 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:21.394725 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/786969b9-72c9-4b56-91e4-97d255bbe6e4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/786969b9-72c9-4b56-91e4-97d255bbe6e4-tmp-dir\") pod \"786969b9-72c9-4b56-91e4-97d255bbe6e4\" (UID: \"786969b9-72c9-4b56-91e4-97d255bbe6e4\") " Nov 03 02:39:21.394927 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:21.394820 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/786969b9-72c9-4b56-91e4-97d255bbe6e4-home\" (UniqueName: \"kubernetes.io/empty-dir/786969b9-72c9-4b56-91e4-97d255bbe6e4-home\") pod \"786969b9-72c9-4b56-91e4-97d255bbe6e4\" (UID: \"786969b9-72c9-4b56-91e4-97d255bbe6e4\") " Nov 03 02:39:21.394927 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:21.394861 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/786969b9-72c9-4b56-91e4-97d255bbe6e4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/786969b9-72c9-4b56-91e4-97d255bbe6e4-model-cache\") pod \"786969b9-72c9-4b56-91e4-97d255bbe6e4\" (UID: \"786969b9-72c9-4b56-91e4-97d255bbe6e4\") " Nov 03 02:39:21.394927 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:21.394916 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/786969b9-72c9-4b56-91e4-97d255bbe6e4-tls-certs\" (UniqueName: \"kubernetes.io/secret/786969b9-72c9-4b56-91e4-97d255bbe6e4-tls-certs\") pod \"786969b9-72c9-4b56-91e4-97d255bbe6e4\" (UID: \"786969b9-72c9-4b56-91e4-97d255bbe6e4\") " Nov 03 02:39:21.395158 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:21.395105 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/786969b9-72c9-4b56-91e4-97d255bbe6e4-home" pod "786969b9-72c9-4b56-91e4-97d255bbe6e4" (UID: "786969b9-72c9-4b56-91e4-97d255bbe6e4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:39:21.395219 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:21.395165 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/786969b9-72c9-4b56-91e4-97d255bbe6e4-model-cache" pod "786969b9-72c9-4b56-91e4-97d255bbe6e4" (UID: "786969b9-72c9-4b56-91e4-97d255bbe6e4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:39:21.395219 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:21.395105 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/786969b9-72c9-4b56-91e4-97d255bbe6e4-tmp-dir" pod "786969b9-72c9-4b56-91e4-97d255bbe6e4" (UID: "786969b9-72c9-4b56-91e4-97d255bbe6e4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:39:21.395515 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:21.395489 5437 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/786969b9-72c9-4b56-91e4-97d255bbe6e4-tmp-dir\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:39:21.395515 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:21.395517 5437 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/786969b9-72c9-4b56-91e4-97d255bbe6e4-home\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:39:21.395723 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:21.395532 5437 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/786969b9-72c9-4b56-91e4-97d255bbe6e4-model-cache\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:39:21.396992 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:21.396967 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/786969b9-72c9-4b56-91e4-97d255bbe6e4-dshm" pod "786969b9-72c9-4b56-91e4-97d255bbe6e4" (UID: "786969b9-72c9-4b56-91e4-97d255bbe6e4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:39:21.397075 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:21.397016 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/786969b9-72c9-4b56-91e4-97d255bbe6e4-tls-certs" pod "786969b9-72c9-4b56-91e4-97d255bbe6e4" (UID: "786969b9-72c9-4b56-91e4-97d255bbe6e4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:39:21.397075 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:21.397026 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/786969b9-72c9-4b56-91e4-97d255bbe6e4-kube-api-access-v8gpr" pod "786969b9-72c9-4b56-91e4-97d255bbe6e4" (UID: "786969b9-72c9-4b56-91e4-97d255bbe6e4"). InnerVolumeSpecName "kube-api-access-v8gpr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:39:21.496822 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:21.496789 5437 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/786969b9-72c9-4b56-91e4-97d255bbe6e4-tls-certs\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:39:21.496822 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:21.496818 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8gpr\" (UniqueName: \"kubernetes.io/projected/786969b9-72c9-4b56-91e4-97d255bbe6e4-kube-api-access-v8gpr\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:39:21.496822 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:21.496828 5437 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/786969b9-72c9-4b56-91e4-97d255bbe6e4-dshm\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:39:21.501540 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:21.501499 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:39:21.525442 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:21.525403 5437 generic.go:356] "Generic (PLEG): container finished" podID="786969b9-72c9-4b56-91e4-97d255bbe6e4" containerID="6b5fddaf6969ccbec71ef3f69e35af11ec74b017e065ac02ed99bff495aa6670" exitCode=0 Nov 03 02:39:21.525637 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:21.525482 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-98a394/rate-test-model-98a394-kserve-6d7884c4c7-gp5g4" event={"ID":"786969b9-72c9-4b56-91e4-97d255bbe6e4","Type":"ContainerDied","Data":"6b5fddaf6969ccbec71ef3f69e35af11ec74b017e065ac02ed99bff495aa6670"} Nov 03 02:39:21.525637 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:21.525514 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-98a394/rate-test-model-98a394-kserve-6d7884c4c7-gp5g4" Nov 03 02:39:21.525637 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:21.525532 5437 scope.go:122] "RemoveContainer" containerID="6b5fddaf6969ccbec71ef3f69e35af11ec74b017e065ac02ed99bff495aa6670" Nov 03 02:39:21.525806 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:21.525517 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-98a394/rate-test-model-98a394-kserve-6d7884c4c7-gp5g4" event={"ID":"786969b9-72c9-4b56-91e4-97d255bbe6e4","Type":"ContainerDied","Data":"6ae0482e8289602cfcef45b3298c8df2d7b1eb631c16737495842caca992ba2c"} Nov 03 02:39:21.535511 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:21.535487 5437 scope.go:122] "RemoveContainer" containerID="6b5fddaf6969ccbec71ef3f69e35af11ec74b017e065ac02ed99bff495aa6670" Nov 03 02:39:21.535839 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:39:21.535812 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b5fddaf6969ccbec71ef3f69e35af11ec74b017e065ac02ed99bff495aa6670\": container with ID starting with 6b5fddaf6969ccbec71ef3f69e35af11ec74b017e065ac02ed99bff495aa6670 not found: ID does not exist" containerID="6b5fddaf6969ccbec71ef3f69e35af11ec74b017e065ac02ed99bff495aa6670" Nov 03 02:39:21.535930 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:21.535846 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b5fddaf6969ccbec71ef3f69e35af11ec74b017e065ac02ed99bff495aa6670"} err="failed to get container status \"6b5fddaf6969ccbec71ef3f69e35af11ec74b017e065ac02ed99bff495aa6670\": rpc error: code = NotFound desc = could not find container \"6b5fddaf6969ccbec71ef3f69e35af11ec74b017e065ac02ed99bff495aa6670\": container with ID starting with 6b5fddaf6969ccbec71ef3f69e35af11ec74b017e065ac02ed99bff495aa6670 not found: ID does not exist" Nov 03 02:39:21.699300 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:21.699189 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-98a394/rate-test-model-98a394-kserve-6d7884c4c7-gp5g4"] Nov 03 02:39:21.703684 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:21.703637 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-98a394/rate-test-model-98a394-kserve-6d7884c4c7-gp5g4"] Nov 03 02:39:22.088834 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:22.088801 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="786969b9-72c9-4b56-91e4-97d255bbe6e4" path="/var/lib/kubelet/pods/786969b9-72c9-4b56-91e4-97d255bbe6e4/volumes" Nov 03 02:39:25.070434 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:25.070390 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:39:29.015144 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:29.015103 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:39:29.893412 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:29.893351 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:39:30.299955 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:30.299912 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:39:30.847609 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:30.847562 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:39:33.384003 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:33.383974 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w0-f0391a/sub-test-model-f0391a-kserve-dd566f79c-sqxqq"] Nov 03 02:39:33.384589 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:33.384570 5437 cpu_manager.go:414] "containerMap: removing container" podUID="c3d40c05-1fca-4cad-873c-cc68bc420627" containerName="payload-pre-processing" Nov 03 02:39:33.384642 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:33.384590 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3d40c05-1fca-4cad-873c-cc68bc420627" containerName="payload-pre-processing" Nov 03 02:39:33.384642 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:33.384604 5437 cpu_manager.go:414] "containerMap: removing container" podUID="ff3f6c0f-5c72-4897-ac51-f9cef66c9f95" containerName="main" Nov 03 02:39:33.384642 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:33.384610 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff3f6c0f-5c72-4897-ac51-f9cef66c9f95" containerName="main" Nov 03 02:39:33.384642 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:33.384616 5437 cpu_manager.go:414] "containerMap: removing container" podUID="295d5843-32eb-47f1-b942-a71a98cb2ca7" containerName="payload-processing" Nov 03 02:39:33.384642 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:33.384621 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="295d5843-32eb-47f1-b942-a71a98cb2ca7" containerName="payload-processing" Nov 03 02:39:33.384642 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:33.384633 5437 cpu_manager.go:414] "containerMap: removing container" podUID="f622668a-42fc-4471-9b7f-1166924c2769" containerName="payload-processing" Nov 03 02:39:33.384642 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:33.384638 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f622668a-42fc-4471-9b7f-1166924c2769" containerName="payload-processing" Nov 03 02:39:33.384876 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:33.384657 5437 cpu_manager.go:414] "containerMap: removing container" podUID="250a28b0-9884-43a0-9991-61fb8e3a5be4" containerName="istio-proxy" Nov 03 02:39:33.384876 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:33.384662 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="250a28b0-9884-43a0-9991-61fb8e3a5be4" containerName="istio-proxy" Nov 03 02:39:33.384876 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:33.384675 5437 cpu_manager.go:414] "containerMap: removing container" podUID="e1907086-a8a1-40a5-9a85-8b9d9ea21e40" containerName="main" Nov 03 02:39:33.384876 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:33.384680 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1907086-a8a1-40a5-9a85-8b9d9ea21e40" containerName="main" Nov 03 02:39:33.384876 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:33.384697 5437 cpu_manager.go:414] "containerMap: removing container" podUID="4cd50b12-718f-4a10-a4f5-16a17fc90342" containerName="payload-processing" Nov 03 02:39:33.384876 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:33.384702 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cd50b12-718f-4a10-a4f5-16a17fc90342" containerName="payload-processing" Nov 03 02:39:33.384876 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:33.384712 5437 cpu_manager.go:414] "containerMap: removing container" podUID="786969b9-72c9-4b56-91e4-97d255bbe6e4" containerName="main" Nov 03 02:39:33.384876 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:33.384716 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="786969b9-72c9-4b56-91e4-97d255bbe6e4" containerName="main" Nov 03 02:39:33.384876 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:33.384802 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cd50b12-718f-4a10-a4f5-16a17fc90342" containerName="payload-processing" Nov 03 02:39:33.384876 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:33.384810 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="786969b9-72c9-4b56-91e4-97d255bbe6e4" containerName="main" Nov 03 02:39:33.384876 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:33.384821 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3d40c05-1fca-4cad-873c-cc68bc420627" containerName="payload-pre-processing" Nov 03 02:39:33.384876 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:33.384827 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff3f6c0f-5c72-4897-ac51-f9cef66c9f95" containerName="main" Nov 03 02:39:33.384876 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:33.384832 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="295d5843-32eb-47f1-b942-a71a98cb2ca7" containerName="payload-processing" Nov 03 02:39:33.384876 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:33.384839 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="f622668a-42fc-4471-9b7f-1166924c2769" containerName="payload-processing" Nov 03 02:39:33.384876 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:33.384846 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="250a28b0-9884-43a0-9991-61fb8e3a5be4" containerName="istio-proxy" Nov 03 02:39:33.384876 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:33.384851 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1907086-a8a1-40a5-9a85-8b9d9ea21e40" containerName="main" Nov 03 02:39:33.389678 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:33.389655 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-f0391a/sub-test-model-f0391a-kserve-dd566f79c-sqxqq" Nov 03 02:39:33.443009 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:33.442974 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-f0391a/sub-test-model-f0391a-kserve-dd566f79c-sqxqq"] Nov 03 02:39:33.491928 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:33.491894 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-f0391a\"/\"openshift-service-ca.crt\"" Nov 03 02:39:33.492164 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:33.492027 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-f0391a\"/\"kube-root-ca.crt\"" Nov 03 02:39:33.492164 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:33.492119 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-f0391a\"/\"default-dockercfg-lmkjf\"" Nov 03 02:39:33.492420 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:33.492399 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:39:33.495326 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:33.495310 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-f0391a\"/\"sub-test-model-f0391a-kserve-self-signed-certs\"" Nov 03 02:39:33.537614 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:33.537581 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4467722-3cdf-4e97-b559-2e5e5b0b62db-home\") pod \"sub-test-model-f0391a-kserve-dd566f79c-sqxqq\" (UID: \"f4467722-3cdf-4e97-b559-2e5e5b0b62db\") " pod="ai-tenant-e2e-shared-b-w0-f0391a/sub-test-model-f0391a-kserve-dd566f79c-sqxqq" Nov 03 02:39:33.537782 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:33.537638 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4467722-3cdf-4e97-b559-2e5e5b0b62db-tls-certs\") pod \"sub-test-model-f0391a-kserve-dd566f79c-sqxqq\" (UID: \"f4467722-3cdf-4e97-b559-2e5e5b0b62db\") " pod="ai-tenant-e2e-shared-b-w0-f0391a/sub-test-model-f0391a-kserve-dd566f79c-sqxqq" Nov 03 02:39:33.537782 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:33.537695 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4467722-3cdf-4e97-b559-2e5e5b0b62db-model-cache\") pod \"sub-test-model-f0391a-kserve-dd566f79c-sqxqq\" (UID: \"f4467722-3cdf-4e97-b559-2e5e5b0b62db\") " pod="ai-tenant-e2e-shared-b-w0-f0391a/sub-test-model-f0391a-kserve-dd566f79c-sqxqq" Nov 03 02:39:33.537782 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:33.537730 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4467722-3cdf-4e97-b559-2e5e5b0b62db-tmp-dir\") pod \"sub-test-model-f0391a-kserve-dd566f79c-sqxqq\" (UID: \"f4467722-3cdf-4e97-b559-2e5e5b0b62db\") " pod="ai-tenant-e2e-shared-b-w0-f0391a/sub-test-model-f0391a-kserve-dd566f79c-sqxqq" Nov 03 02:39:33.537782 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:33.537756 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ks6z\" (UniqueName: \"kubernetes.io/projected/f4467722-3cdf-4e97-b559-2e5e5b0b62db-kube-api-access-2ks6z\") pod \"sub-test-model-f0391a-kserve-dd566f79c-sqxqq\" (UID: \"f4467722-3cdf-4e97-b559-2e5e5b0b62db\") " pod="ai-tenant-e2e-shared-b-w0-f0391a/sub-test-model-f0391a-kserve-dd566f79c-sqxqq" Nov 03 02:39:33.537945 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:33.537788 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4467722-3cdf-4e97-b559-2e5e5b0b62db-dshm\") pod \"sub-test-model-f0391a-kserve-dd566f79c-sqxqq\" (UID: \"f4467722-3cdf-4e97-b559-2e5e5b0b62db\") " pod="ai-tenant-e2e-shared-b-w0-f0391a/sub-test-model-f0391a-kserve-dd566f79c-sqxqq" Nov 03 02:39:33.639330 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:33.639233 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4467722-3cdf-4e97-b559-2e5e5b0b62db-tmp-dir\") pod \"sub-test-model-f0391a-kserve-dd566f79c-sqxqq\" (UID: \"f4467722-3cdf-4e97-b559-2e5e5b0b62db\") " pod="ai-tenant-e2e-shared-b-w0-f0391a/sub-test-model-f0391a-kserve-dd566f79c-sqxqq" Nov 03 02:39:33.639330 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:33.639282 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ks6z\" (UniqueName: \"kubernetes.io/projected/f4467722-3cdf-4e97-b559-2e5e5b0b62db-kube-api-access-2ks6z\") pod \"sub-test-model-f0391a-kserve-dd566f79c-sqxqq\" (UID: \"f4467722-3cdf-4e97-b559-2e5e5b0b62db\") " pod="ai-tenant-e2e-shared-b-w0-f0391a/sub-test-model-f0391a-kserve-dd566f79c-sqxqq" Nov 03 02:39:33.639330 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:33.639329 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4467722-3cdf-4e97-b559-2e5e5b0b62db-dshm\") pod \"sub-test-model-f0391a-kserve-dd566f79c-sqxqq\" (UID: \"f4467722-3cdf-4e97-b559-2e5e5b0b62db\") " pod="ai-tenant-e2e-shared-b-w0-f0391a/sub-test-model-f0391a-kserve-dd566f79c-sqxqq" Nov 03 02:39:33.639695 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:33.639486 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4467722-3cdf-4e97-b559-2e5e5b0b62db-home\") pod \"sub-test-model-f0391a-kserve-dd566f79c-sqxqq\" (UID: \"f4467722-3cdf-4e97-b559-2e5e5b0b62db\") " pod="ai-tenant-e2e-shared-b-w0-f0391a/sub-test-model-f0391a-kserve-dd566f79c-sqxqq" Nov 03 02:39:33.639695 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:33.639552 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4467722-3cdf-4e97-b559-2e5e5b0b62db-tls-certs\") pod \"sub-test-model-f0391a-kserve-dd566f79c-sqxqq\" (UID: \"f4467722-3cdf-4e97-b559-2e5e5b0b62db\") " pod="ai-tenant-e2e-shared-b-w0-f0391a/sub-test-model-f0391a-kserve-dd566f79c-sqxqq" Nov 03 02:39:33.639695 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:33.639581 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4467722-3cdf-4e97-b559-2e5e5b0b62db-model-cache\") pod \"sub-test-model-f0391a-kserve-dd566f79c-sqxqq\" (UID: \"f4467722-3cdf-4e97-b559-2e5e5b0b62db\") " pod="ai-tenant-e2e-shared-b-w0-f0391a/sub-test-model-f0391a-kserve-dd566f79c-sqxqq" Nov 03 02:39:33.639850 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:33.639773 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4467722-3cdf-4e97-b559-2e5e5b0b62db-tmp-dir\") pod \"sub-test-model-f0391a-kserve-dd566f79c-sqxqq\" (UID: \"f4467722-3cdf-4e97-b559-2e5e5b0b62db\") " pod="ai-tenant-e2e-shared-b-w0-f0391a/sub-test-model-f0391a-kserve-dd566f79c-sqxqq" Nov 03 02:39:33.639889 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:33.639836 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4467722-3cdf-4e97-b559-2e5e5b0b62db-home\") pod \"sub-test-model-f0391a-kserve-dd566f79c-sqxqq\" (UID: \"f4467722-3cdf-4e97-b559-2e5e5b0b62db\") " pod="ai-tenant-e2e-shared-b-w0-f0391a/sub-test-model-f0391a-kserve-dd566f79c-sqxqq" Nov 03 02:39:33.639889 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:33.639883 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4467722-3cdf-4e97-b559-2e5e5b0b62db-model-cache\") pod \"sub-test-model-f0391a-kserve-dd566f79c-sqxqq\" (UID: \"f4467722-3cdf-4e97-b559-2e5e5b0b62db\") " pod="ai-tenant-e2e-shared-b-w0-f0391a/sub-test-model-f0391a-kserve-dd566f79c-sqxqq" Nov 03 02:39:33.641702 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:33.641680 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4467722-3cdf-4e97-b559-2e5e5b0b62db-dshm\") pod \"sub-test-model-f0391a-kserve-dd566f79c-sqxqq\" (UID: \"f4467722-3cdf-4e97-b559-2e5e5b0b62db\") " pod="ai-tenant-e2e-shared-b-w0-f0391a/sub-test-model-f0391a-kserve-dd566f79c-sqxqq" Nov 03 02:39:33.642082 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:33.642057 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4467722-3cdf-4e97-b559-2e5e5b0b62db-tls-certs\") pod \"sub-test-model-f0391a-kserve-dd566f79c-sqxqq\" (UID: \"f4467722-3cdf-4e97-b559-2e5e5b0b62db\") " pod="ai-tenant-e2e-shared-b-w0-f0391a/sub-test-model-f0391a-kserve-dd566f79c-sqxqq" Nov 03 02:39:33.701810 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:33.701780 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ks6z\" (UniqueName: \"kubernetes.io/projected/f4467722-3cdf-4e97-b559-2e5e5b0b62db-kube-api-access-2ks6z\") pod \"sub-test-model-f0391a-kserve-dd566f79c-sqxqq\" (UID: \"f4467722-3cdf-4e97-b559-2e5e5b0b62db\") " pod="ai-tenant-e2e-shared-b-w0-f0391a/sub-test-model-f0391a-kserve-dd566f79c-sqxqq" Nov 03 02:39:33.704612 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:33.704589 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-f0391a/sub-test-model-f0391a-kserve-dd566f79c-sqxqq" Nov 03 02:39:34.016161 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:34.016074 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:39:34.075581 ip-10-0-136-16 kubenswrapper[5437]: W1103 02:39:34.075539 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf4467722_3cdf_4e97_b559_2e5e5b0b62db.slice/crio-a3b8b06caf8cf60981210b7c663016ed557993fdce98cc433c18a40ea84a74a0 WatchSource:0}: Error finding container a3b8b06caf8cf60981210b7c663016ed557993fdce98cc433c18a40ea84a74a0: Status 404 returned error can't find the container with id a3b8b06caf8cf60981210b7c663016ed557993fdce98cc433c18a40ea84a74a0 Nov 03 02:39:34.078524 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:34.078497 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-f0391a/sub-test-model-f0391a-kserve-dd566f79c-sqxqq"] Nov 03 02:39:34.601447 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:34.601401 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-f0391a/sub-test-model-f0391a-kserve-dd566f79c-sqxqq" event={"ID":"f4467722-3cdf-4e97-b559-2e5e5b0b62db","Type":"ContainerStarted","Data":"46778fc78ed4eb4f02f6c0c1ae18bdd45c4bb543440778be2f44079d5a51a828"} Nov 03 02:39:34.601447 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:34.601444 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-f0391a/sub-test-model-f0391a-kserve-dd566f79c-sqxqq" event={"ID":"f4467722-3cdf-4e97-b559-2e5e5b0b62db","Type":"ContainerStarted","Data":"a3b8b06caf8cf60981210b7c663016ed557993fdce98cc433c18a40ea84a74a0"} Nov 03 02:39:34.705229 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:34.705173 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w0-f0391a/sub-test-model-f0391a-kserve-dd566f79c-sqxqq" Nov 03 02:39:34.724159 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:34.724099 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w0-f0391a/sub-test-model-f0391a-kserve-dd566f79c-sqxqq" podStartSLOduration=1.724085692 podStartE2EDuration="1.724085692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:39:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:39:34.722543545 +0000 UTC m=+4856.752025841" watchObservedRunningTime="2026-11-03 02:39:34.724085692 +0000 UTC m=+4856.753567986" Nov 03 02:39:35.682637 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:35.682595 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:39:36.751787 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:36.751750 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:39:36.878692 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:36.878646 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-98a394-67d66f5849-ktn8h"] Nov 03 02:39:37.274014 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:37.273978 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:39:37.474619 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:37.474580 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:39:40.722537 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:40.722497 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:39:41.178183 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:41.178140 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:39:41.879485 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:41.879434 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-98a394-67d66f5849-ktn8h" podUID="85af418f-1d3c-44dd-803b-fd73c7e18b0d" containerName="payload-processing" containerID="cri-o://3230d5692cbf8f6be880387103819069e94c4908bf678210dccb91ba64341d97" gracePeriod=25 Nov 03 02:39:42.255889 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:42.255794 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:39:42.731702 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:42.731659 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:39:42.935569 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:42.935532 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:39:43.611549 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:43.611513 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:39:43.705242 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:43.705203 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w0-f0391a/sub-test-model-f0391a-kserve-dd566f79c-sqxqq" Nov 03 02:39:43.718067 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:43.718036 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w0-f0391a/sub-test-model-f0391a-kserve-dd566f79c-sqxqq" Nov 03 02:39:44.034923 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:39:44.034831 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 02:39:44.668598 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:44.668558 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w0-f0391a/sub-test-model-f0391a-kserve-dd566f79c-sqxqq" Nov 03 02:39:44.838608 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:44.838570 5437 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-98a394-67d66f5849-ktn8h" podUID="85af418f-1d3c-44dd-803b-fd73c7e18b0d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 02:39:45.839300 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:45.839267 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:39:49.650033 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:49.649949 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:39:50.490108 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:50.490069 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-d9576a-5556d5f859-sr8mk"] Nov 03 02:39:50.796529 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:50.796413 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:39:50.994884 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:50.994847 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:39:51.208145 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:51.208104 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:39:54.838471 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:54.838424 5437 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-98a394-67d66f5849-ktn8h" podUID="85af418f-1d3c-44dd-803b-fd73c7e18b0d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 02:39:55.421149 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:55.421110 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:39:55.490910 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:55.490840 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-d9576a-5556d5f859-sr8mk" podUID="6050630e-79eb-49c9-bb97-4cd9e9309ffa" containerName="payload-processing" containerID="cri-o://ceb93c34598d433b93aa473eb655c7a5941ee64b09c5002f23159d560191ec41" gracePeriod=25 Nov 03 02:39:55.966484 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:55.966441 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:39:59.970613 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:59.970578 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-434b6a-7d4c599fb6-t9vcs"] Nov 03 02:39:59.971153 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:39:59.970787 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-434b6a-7d4c599fb6-t9vcs" podUID="21b04f72-d240-4ebb-860b-c80607e0d94d" containerName="maas-api" containerID="cri-o://e71e8296b2f8c8956645d07aa0e055907cba9399d9a49f8f25ad5808ccfbc03e" gracePeriod=30 Nov 03 02:40:00.126103 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:00.126064 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:40:00.243636 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:00.243608 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-434b6a-7d4c599fb6-t9vcs" Nov 03 02:40:00.348186 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:00.348149 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21b04f72-d240-4ebb-860b-c80607e0d94d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21b04f72-d240-4ebb-860b-c80607e0d94d-maas-api-tls\") pod \"21b04f72-d240-4ebb-860b-c80607e0d94d\" (UID: \"21b04f72-d240-4ebb-860b-c80607e0d94d\") " Nov 03 02:40:00.348394 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:00.348313 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21b04f72-d240-4ebb-860b-c80607e0d94d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21b04f72-d240-4ebb-860b-c80607e0d94d-openshift-service-ca\") pod \"21b04f72-d240-4ebb-860b-c80607e0d94d\" (UID: \"21b04f72-d240-4ebb-860b-c80607e0d94d\") " Nov 03 02:40:00.348394 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:00.348378 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21b04f72-d240-4ebb-860b-c80607e0d94d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/21b04f72-d240-4ebb-860b-c80607e0d94d-metrics-certs\") pod \"21b04f72-d240-4ebb-860b-c80607e0d94d\" (UID: \"21b04f72-d240-4ebb-860b-c80607e0d94d\") " Nov 03 02:40:00.348554 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:00.348504 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21b04f72-d240-4ebb-860b-c80607e0d94d-kube-api-access-tvndr\" (UniqueName: \"kubernetes.io/projected/21b04f72-d240-4ebb-860b-c80607e0d94d-kube-api-access-tvndr\") pod \"21b04f72-d240-4ebb-860b-c80607e0d94d\" (UID: \"21b04f72-d240-4ebb-860b-c80607e0d94d\") " Nov 03 02:40:00.348756 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:00.348728 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21b04f72-d240-4ebb-860b-c80607e0d94d-openshift-service-ca" pod "21b04f72-d240-4ebb-860b-c80607e0d94d" (UID: "21b04f72-d240-4ebb-860b-c80607e0d94d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:40:00.349014 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:00.348984 5437 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21b04f72-d240-4ebb-860b-c80607e0d94d-openshift-service-ca\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:40:00.350658 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:00.350619 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21b04f72-d240-4ebb-860b-c80607e0d94d-metrics-certs" pod "21b04f72-d240-4ebb-860b-c80607e0d94d" (UID: "21b04f72-d240-4ebb-860b-c80607e0d94d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:40:00.350869 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:00.350848 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21b04f72-d240-4ebb-860b-c80607e0d94d-maas-api-tls" pod "21b04f72-d240-4ebb-860b-c80607e0d94d" (UID: "21b04f72-d240-4ebb-860b-c80607e0d94d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:40:00.351346 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:00.351309 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21b04f72-d240-4ebb-860b-c80607e0d94d-kube-api-access-tvndr" pod "21b04f72-d240-4ebb-860b-c80607e0d94d" (UID: "21b04f72-d240-4ebb-860b-c80607e0d94d"). InnerVolumeSpecName "kube-api-access-tvndr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:40:00.450621 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:00.450588 5437 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21b04f72-d240-4ebb-860b-c80607e0d94d-metrics-certs\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:40:00.450621 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:00.450616 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvndr\" (UniqueName: \"kubernetes.io/projected/21b04f72-d240-4ebb-860b-c80607e0d94d-kube-api-access-tvndr\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:40:00.450621 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:00.450626 5437 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21b04f72-d240-4ebb-860b-c80607e0d94d-maas-api-tls\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:40:00.752942 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:00.752905 5437 generic.go:356] "Generic (PLEG): container finished" podID="21b04f72-d240-4ebb-860b-c80607e0d94d" containerID="e71e8296b2f8c8956645d07aa0e055907cba9399d9a49f8f25ad5808ccfbc03e" exitCode=0 Nov 03 02:40:00.753134 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:00.752958 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-434b6a-7d4c599fb6-t9vcs" Nov 03 02:40:00.753134 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:00.752981 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-434b6a-7d4c599fb6-t9vcs" event={"ID":"21b04f72-d240-4ebb-860b-c80607e0d94d","Type":"ContainerDied","Data":"e71e8296b2f8c8956645d07aa0e055907cba9399d9a49f8f25ad5808ccfbc03e"} Nov 03 02:40:00.753134 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:00.753037 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-434b6a-7d4c599fb6-t9vcs" event={"ID":"21b04f72-d240-4ebb-860b-c80607e0d94d","Type":"ContainerDied","Data":"26192d45480c586ae09589db229a62f22e3c92de5d20c7ed365199c2536dceee"} Nov 03 02:40:00.753134 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:00.753059 5437 scope.go:122] "RemoveContainer" containerID="e71e8296b2f8c8956645d07aa0e055907cba9399d9a49f8f25ad5808ccfbc03e" Nov 03 02:40:00.763077 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:00.763061 5437 scope.go:122] "RemoveContainer" containerID="e71e8296b2f8c8956645d07aa0e055907cba9399d9a49f8f25ad5808ccfbc03e" Nov 03 02:40:00.763352 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:40:00.763333 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e71e8296b2f8c8956645d07aa0e055907cba9399d9a49f8f25ad5808ccfbc03e\": container with ID starting with e71e8296b2f8c8956645d07aa0e055907cba9399d9a49f8f25ad5808ccfbc03e not found: ID does not exist" containerID="e71e8296b2f8c8956645d07aa0e055907cba9399d9a49f8f25ad5808ccfbc03e" Nov 03 02:40:00.763411 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:00.763360 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e71e8296b2f8c8956645d07aa0e055907cba9399d9a49f8f25ad5808ccfbc03e"} err="failed to get container status \"e71e8296b2f8c8956645d07aa0e055907cba9399d9a49f8f25ad5808ccfbc03e\": rpc error: code = NotFound desc = could not find container \"e71e8296b2f8c8956645d07aa0e055907cba9399d9a49f8f25ad5808ccfbc03e\": container with ID starting with e71e8296b2f8c8956645d07aa0e055907cba9399d9a49f8f25ad5808ccfbc03e not found: ID does not exist" Nov 03 02:40:00.922444 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:00.922408 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-434b6a-7d4c599fb6-t9vcs"] Nov 03 02:40:00.927010 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:00.926984 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-434b6a-7d4c599fb6-t9vcs"] Nov 03 02:40:01.721550 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:01.721514 5437 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-d9576a-5556d5f859-sr8mk" podUID="6050630e-79eb-49c9-bb97-4cd9e9309ffa" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 02:40:01.788165 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:01.788129 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:40:02.089944 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:02.089910 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21b04f72-d240-4ebb-860b-c80607e0d94d" path="/var/lib/kubelet/pods/21b04f72-d240-4ebb-860b-c80607e0d94d/volumes" Nov 03 02:40:02.142174 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:02.142146 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-98a394-67d66f5849-ktn8h" Nov 03 02:40:02.272877 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:02.272782 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85af418f-1d3c-44dd-803b-fd73c7e18b0d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85af418f-1d3c-44dd-803b-fd73c7e18b0d-plugins-config-volume\") pod \"85af418f-1d3c-44dd-803b-fd73c7e18b0d\" (UID: \"85af418f-1d3c-44dd-803b-fd73c7e18b0d\") " Nov 03 02:40:02.272877 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:02.272833 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85af418f-1d3c-44dd-803b-fd73c7e18b0d-kube-api-access-nwlz2\" (UniqueName: \"kubernetes.io/projected/85af418f-1d3c-44dd-803b-fd73c7e18b0d-kube-api-access-nwlz2\") pod \"85af418f-1d3c-44dd-803b-fd73c7e18b0d\" (UID: \"85af418f-1d3c-44dd-803b-fd73c7e18b0d\") " Nov 03 02:40:02.273242 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:02.273213 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85af418f-1d3c-44dd-803b-fd73c7e18b0d-plugins-config-volume" pod "85af418f-1d3c-44dd-803b-fd73c7e18b0d" (UID: "85af418f-1d3c-44dd-803b-fd73c7e18b0d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:40:02.275028 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:02.275008 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85af418f-1d3c-44dd-803b-fd73c7e18b0d-kube-api-access-nwlz2" pod "85af418f-1d3c-44dd-803b-fd73c7e18b0d" (UID: "85af418f-1d3c-44dd-803b-fd73c7e18b0d"). InnerVolumeSpecName "kube-api-access-nwlz2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:40:02.374816 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:02.374783 5437 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85af418f-1d3c-44dd-803b-fd73c7e18b0d-plugins-config-volume\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:40:02.374816 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:02.374809 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwlz2\" (UniqueName: \"kubernetes.io/projected/85af418f-1d3c-44dd-803b-fd73c7e18b0d-kube-api-access-nwlz2\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:40:02.765995 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:02.765955 5437 generic.go:356] "Generic (PLEG): container finished" podID="85af418f-1d3c-44dd-803b-fd73c7e18b0d" containerID="3230d5692cbf8f6be880387103819069e94c4908bf678210dccb91ba64341d97" exitCode=0 Nov 03 02:40:02.766436 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:02.766011 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-98a394-67d66f5849-ktn8h" Nov 03 02:40:02.766436 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:02.766033 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-98a394-67d66f5849-ktn8h" event={"ID":"85af418f-1d3c-44dd-803b-fd73c7e18b0d","Type":"ContainerDied","Data":"3230d5692cbf8f6be880387103819069e94c4908bf678210dccb91ba64341d97"} Nov 03 02:40:02.766436 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:02.766093 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-98a394-67d66f5849-ktn8h" event={"ID":"85af418f-1d3c-44dd-803b-fd73c7e18b0d","Type":"ContainerDied","Data":"a2852e30eacd03f073bea0ec3087d1968d076c8541c5b3c2f357adbbc3b857cd"} Nov 03 02:40:02.766436 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:02.766115 5437 scope.go:122] "RemoveContainer" containerID="3230d5692cbf8f6be880387103819069e94c4908bf678210dccb91ba64341d97" Nov 03 02:40:02.776283 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:02.776256 5437 scope.go:122] "RemoveContainer" containerID="3230d5692cbf8f6be880387103819069e94c4908bf678210dccb91ba64341d97" Nov 03 02:40:02.776630 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:40:02.776607 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3230d5692cbf8f6be880387103819069e94c4908bf678210dccb91ba64341d97\": container with ID starting with 3230d5692cbf8f6be880387103819069e94c4908bf678210dccb91ba64341d97 not found: ID does not exist" containerID="3230d5692cbf8f6be880387103819069e94c4908bf678210dccb91ba64341d97" Nov 03 02:40:02.776714 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:02.776639 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3230d5692cbf8f6be880387103819069e94c4908bf678210dccb91ba64341d97"} err="failed to get container status \"3230d5692cbf8f6be880387103819069e94c4908bf678210dccb91ba64341d97\": rpc error: code = NotFound desc = could not find container \"3230d5692cbf8f6be880387103819069e94c4908bf678210dccb91ba64341d97\": container with ID starting with 3230d5692cbf8f6be880387103819069e94c4908bf678210dccb91ba64341d97 not found: ID does not exist" Nov 03 02:40:02.938325 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:02.938285 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-98a394-67d66f5849-ktn8h"] Nov 03 02:40:02.990294 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:02.990256 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-98a394-67d66f5849-ktn8h"] Nov 03 02:40:03.596833 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:03.596793 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:40:04.088291 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:04.088249 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85af418f-1d3c-44dd-803b-fd73c7e18b0d" path="/var/lib/kubelet/pods/85af418f-1d3c-44dd-803b-fd73c7e18b0d/volumes" Nov 03 02:40:04.618155 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:04.618117 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:40:04.818175 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:04.818141 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:40:06.214593 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:06.214558 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:40:06.746572 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:06.746534 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:40:06.987773 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:06.987714 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:40:07.212046 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:07.211996 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:40:08.567731 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:08.567696 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:40:11.100253 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:11.100214 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:40:11.309581 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:11.309535 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:40:11.720628 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:11.720590 5437 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-d9576a-5556d5f859-sr8mk" podUID="6050630e-79eb-49c9-bb97-4cd9e9309ffa" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 02:40:11.909390 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:11.909337 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:40:12.127065 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:12.127034 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-41aa5f/premium-w3-41aa5f-kserve-5fb94fd5fc-gmtkb"] Nov 03 02:40:12.135640 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:12.135610 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-41aa5f/simulator-w3-41aa5f-kserve-586cfbcdd4-gsmfk"] Nov 03 02:40:12.335205 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:12.335171 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:40:14.034938 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:40:14.034902 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 02:40:15.758388 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:15.758295 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-d9576a-5556d5f859-sr8mk" Nov 03 02:40:15.829744 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:15.829709 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6050630e-79eb-49c9-bb97-4cd9e9309ffa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6050630e-79eb-49c9-bb97-4cd9e9309ffa-plugins-config-volume\") pod \"6050630e-79eb-49c9-bb97-4cd9e9309ffa\" (UID: \"6050630e-79eb-49c9-bb97-4cd9e9309ffa\") " Nov 03 02:40:15.829949 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:15.829906 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6050630e-79eb-49c9-bb97-4cd9e9309ffa-kube-api-access-k42x6\" (UniqueName: \"kubernetes.io/projected/6050630e-79eb-49c9-bb97-4cd9e9309ffa-kube-api-access-k42x6\") pod \"6050630e-79eb-49c9-bb97-4cd9e9309ffa\" (UID: \"6050630e-79eb-49c9-bb97-4cd9e9309ffa\") " Nov 03 02:40:15.830089 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:15.830064 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6050630e-79eb-49c9-bb97-4cd9e9309ffa-plugins-config-volume" pod "6050630e-79eb-49c9-bb97-4cd9e9309ffa" (UID: "6050630e-79eb-49c9-bb97-4cd9e9309ffa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:40:15.830266 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:15.830247 5437 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6050630e-79eb-49c9-bb97-4cd9e9309ffa-plugins-config-volume\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:40:15.832088 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:15.832066 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6050630e-79eb-49c9-bb97-4cd9e9309ffa-kube-api-access-k42x6" pod "6050630e-79eb-49c9-bb97-4cd9e9309ffa" (UID: "6050630e-79eb-49c9-bb97-4cd9e9309ffa"). InnerVolumeSpecName "kube-api-access-k42x6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:40:15.852797 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:15.852766 5437 generic.go:356] "Generic (PLEG): container finished" podID="6050630e-79eb-49c9-bb97-4cd9e9309ffa" containerID="ceb93c34598d433b93aa473eb655c7a5941ee64b09c5002f23159d560191ec41" exitCode=0 Nov 03 02:40:15.852937 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:15.852817 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-d9576a-5556d5f859-sr8mk" Nov 03 02:40:15.852937 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:15.852859 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-d9576a-5556d5f859-sr8mk" event={"ID":"6050630e-79eb-49c9-bb97-4cd9e9309ffa","Type":"ContainerDied","Data":"ceb93c34598d433b93aa473eb655c7a5941ee64b09c5002f23159d560191ec41"} Nov 03 02:40:15.853016 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:15.852948 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-d9576a-5556d5f859-sr8mk" event={"ID":"6050630e-79eb-49c9-bb97-4cd9e9309ffa","Type":"ContainerDied","Data":"2e62d452339cc19155c6c71cf6898590f5598946a12a6fd7a68eac23924a5e27"} Nov 03 02:40:15.853016 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:15.852974 5437 scope.go:122] "RemoveContainer" containerID="ceb93c34598d433b93aa473eb655c7a5941ee64b09c5002f23159d560191ec41" Nov 03 02:40:15.862593 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:15.862574 5437 scope.go:122] "RemoveContainer" containerID="ceb93c34598d433b93aa473eb655c7a5941ee64b09c5002f23159d560191ec41" Nov 03 02:40:15.862877 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:40:15.862858 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ceb93c34598d433b93aa473eb655c7a5941ee64b09c5002f23159d560191ec41\": container with ID starting with ceb93c34598d433b93aa473eb655c7a5941ee64b09c5002f23159d560191ec41 not found: ID does not exist" containerID="ceb93c34598d433b93aa473eb655c7a5941ee64b09c5002f23159d560191ec41" Nov 03 02:40:15.862958 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:15.862885 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ceb93c34598d433b93aa473eb655c7a5941ee64b09c5002f23159d560191ec41"} err="failed to get container status \"ceb93c34598d433b93aa473eb655c7a5941ee64b09c5002f23159d560191ec41\": rpc error: code = NotFound desc = could not find container \"ceb93c34598d433b93aa473eb655c7a5941ee64b09c5002f23159d560191ec41\": container with ID starting with ceb93c34598d433b93aa473eb655c7a5941ee64b09c5002f23159d560191ec41 not found: ID does not exist" Nov 03 02:40:15.931944 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:15.931905 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k42x6\" (UniqueName: \"kubernetes.io/projected/6050630e-79eb-49c9-bb97-4cd9e9309ffa-kube-api-access-k42x6\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:40:15.996916 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:15.996879 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-d9576a-5556d5f859-sr8mk"] Nov 03 02:40:16.002649 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:16.002622 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-d9576a-5556d5f859-sr8mk"] Nov 03 02:40:16.089073 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:16.089042 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6050630e-79eb-49c9-bb97-4cd9e9309ffa" path="/var/lib/kubelet/pods/6050630e-79eb-49c9-bb97-4cd9e9309ffa/volumes" Nov 03 02:40:16.682606 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:16.682571 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:40:17.012868 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:17.012766 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:40:17.211808 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:17.211758 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:40:18.645405 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:18.645351 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:40:21.473630 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:21.473594 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:40:24.714309 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:24.714270 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:40:25.535669 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:25.535631 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:40:25.684331 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:25.684295 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-98a394-d995767bd-68hkm"] Nov 03 02:40:25.684696 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:25.684569 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-98a394-d995767bd-68hkm" podUID="bcc4bb00-2f9e-4f38-a331-f725729873f0" containerName="maas-api" containerID="cri-o://23f4373d4bdc45e4e6e34f427c8f63eb078cb18e181e48a00bd191cf3b4877cd" gracePeriod=30 Nov 03 02:40:25.740987 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:25.740951 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:40:25.909600 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:25.909566 5437 generic.go:356] "Generic (PLEG): container finished" podID="bcc4bb00-2f9e-4f38-a331-f725729873f0" containerID="23f4373d4bdc45e4e6e34f427c8f63eb078cb18e181e48a00bd191cf3b4877cd" exitCode=0 Nov 03 02:40:25.909826 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:25.909612 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-98a394-d995767bd-68hkm" event={"ID":"bcc4bb00-2f9e-4f38-a331-f725729873f0","Type":"ContainerDied","Data":"23f4373d4bdc45e4e6e34f427c8f63eb078cb18e181e48a00bd191cf3b4877cd"} Nov 03 02:40:25.946689 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:25.946649 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:40:25.950509 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:25.950487 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-98a394-d995767bd-68hkm" Nov 03 02:40:26.045751 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:26.045716 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcc4bb00-2f9e-4f38-a331-f725729873f0-kube-api-access-jgq28\" (UniqueName: \"kubernetes.io/projected/bcc4bb00-2f9e-4f38-a331-f725729873f0-kube-api-access-jgq28\") pod \"bcc4bb00-2f9e-4f38-a331-f725729873f0\" (UID: \"bcc4bb00-2f9e-4f38-a331-f725729873f0\") " Nov 03 02:40:26.045953 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:26.045773 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bcc4bb00-2f9e-4f38-a331-f725729873f0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcc4bb00-2f9e-4f38-a331-f725729873f0-metrics-certs\") pod \"bcc4bb00-2f9e-4f38-a331-f725729873f0\" (UID: \"bcc4bb00-2f9e-4f38-a331-f725729873f0\") " Nov 03 02:40:26.045953 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:26.045816 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bcc4bb00-2f9e-4f38-a331-f725729873f0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcc4bb00-2f9e-4f38-a331-f725729873f0-openshift-service-ca\") pod \"bcc4bb00-2f9e-4f38-a331-f725729873f0\" (UID: \"bcc4bb00-2f9e-4f38-a331-f725729873f0\") " Nov 03 02:40:26.046084 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:26.045974 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bcc4bb00-2f9e-4f38-a331-f725729873f0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bcc4bb00-2f9e-4f38-a331-f725729873f0-maas-api-tls\") pod \"bcc4bb00-2f9e-4f38-a331-f725729873f0\" (UID: \"bcc4bb00-2f9e-4f38-a331-f725729873f0\") " Nov 03 02:40:26.046231 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:26.046198 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcc4bb00-2f9e-4f38-a331-f725729873f0-openshift-service-ca" pod "bcc4bb00-2f9e-4f38-a331-f725729873f0" (UID: "bcc4bb00-2f9e-4f38-a331-f725729873f0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:40:26.046494 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:26.046475 5437 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcc4bb00-2f9e-4f38-a331-f725729873f0-openshift-service-ca\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:40:26.048094 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:26.048028 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bcc4bb00-2f9e-4f38-a331-f725729873f0-maas-api-tls" pod "bcc4bb00-2f9e-4f38-a331-f725729873f0" (UID: "bcc4bb00-2f9e-4f38-a331-f725729873f0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:40:26.048094 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:26.048048 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bcc4bb00-2f9e-4f38-a331-f725729873f0-metrics-certs" pod "bcc4bb00-2f9e-4f38-a331-f725729873f0" (UID: "bcc4bb00-2f9e-4f38-a331-f725729873f0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:40:26.048214 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:26.048138 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcc4bb00-2f9e-4f38-a331-f725729873f0-kube-api-access-jgq28" pod "bcc4bb00-2f9e-4f38-a331-f725729873f0" (UID: "bcc4bb00-2f9e-4f38-a331-f725729873f0"). InnerVolumeSpecName "kube-api-access-jgq28". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:40:26.147806 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:26.147770 5437 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcc4bb00-2f9e-4f38-a331-f725729873f0-metrics-certs\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:40:26.147806 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:26.147800 5437 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bcc4bb00-2f9e-4f38-a331-f725729873f0-maas-api-tls\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:40:26.147806 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:26.147810 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgq28\" (UniqueName: \"kubernetes.io/projected/bcc4bb00-2f9e-4f38-a331-f725729873f0-kube-api-access-jgq28\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:40:26.920073 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:26.918979 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-98a394-d995767bd-68hkm" event={"ID":"bcc4bb00-2f9e-4f38-a331-f725729873f0","Type":"ContainerDied","Data":"e2e8545a1016d428f7c486bf343c9afcb40f46659f5c8078bdbf368a5b1a192b"} Nov 03 02:40:26.920073 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:26.919070 5437 scope.go:122] "RemoveContainer" containerID="23f4373d4bdc45e4e6e34f427c8f63eb078cb18e181e48a00bd191cf3b4877cd" Nov 03 02:40:26.920073 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:26.919000 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-98a394-d995767bd-68hkm" Nov 03 02:40:27.009358 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:27.009323 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:40:27.093249 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:27.093206 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-98a394-d995767bd-68hkm"] Nov 03 02:40:27.144671 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:27.144634 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-98a394-d995767bd-68hkm"] Nov 03 02:40:27.148766 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:27.148726 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-41aa5f/premium-w3-41aa5f-kserve-5fb94fd5fc-gmtkb" podUID="c46dca02-6036-46de-b0f3-df796d231f62" containerName="main" containerID="cri-o://a4bacbf873e67a41775fb4df36c84815925b00fb28290cd91cbcc237f594249f" gracePeriod=45 Nov 03 02:40:27.155626 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:27.155590 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-41aa5f/simulator-w3-41aa5f-kserve-586cfbcdd4-gsmfk" podUID="8a87f809-0b26-480e-bb37-ff234a40c5c5" containerName="main" containerID="cri-o://ff1fece57a2d906fc223c5efd375595ffd5ec9aff94298218b650bd18d6aafcb" gracePeriod=45 Nov 03 02:40:27.250242 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:27.250199 5437 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-41aa5f/premium-w3-41aa5f-kserve-5fb94fd5fc-gmtkb" podUID="c46dca02-6036-46de-b0f3-df796d231f62" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": dial tcp 10.134.0.65:8000: connect: connection refused" Nov 03 02:40:27.447292 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:27.447265 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-41aa5f/simulator-w3-41aa5f-kserve-586cfbcdd4-gsmfk" Nov 03 02:40:27.450597 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:27.450571 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-41aa5f/premium-w3-41aa5f-kserve-5fb94fd5fc-gmtkb" Nov 03 02:40:27.496820 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:27.496788 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k42gq"] Nov 03 02:40:27.496986 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:27.496971 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k42gq" podUID="615071b1-f5f9-4d03-90b8-1210c450902d" containerName="istio-proxy" containerID="cri-o://6049cceacbaaff9610f640c7b31bc02e38872a068dce6da6c2782a5c48be6e06" gracePeriod=30 Nov 03 02:40:27.567227 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:27.567065 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a87f809-0b26-480e-bb37-ff234a40c5c5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a87f809-0b26-480e-bb37-ff234a40c5c5-model-cache\") pod \"8a87f809-0b26-480e-bb37-ff234a40c5c5\" (UID: \"8a87f809-0b26-480e-bb37-ff234a40c5c5\") " Nov 03 02:40:27.567227 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:27.567145 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a87f809-0b26-480e-bb37-ff234a40c5c5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a87f809-0b26-480e-bb37-ff234a40c5c5-tmp-dir\") pod \"8a87f809-0b26-480e-bb37-ff234a40c5c5\" (UID: \"8a87f809-0b26-480e-bb37-ff234a40c5c5\") " Nov 03 02:40:27.567227 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:27.567206 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a87f809-0b26-480e-bb37-ff234a40c5c5-tls-certs\" (UniqueName: \"kubernetes.io/secret/8a87f809-0b26-480e-bb37-ff234a40c5c5-tls-certs\") pod \"8a87f809-0b26-480e-bb37-ff234a40c5c5\" (UID: \"8a87f809-0b26-480e-bb37-ff234a40c5c5\") " Nov 03 02:40:27.567715 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:27.567284 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c46dca02-6036-46de-b0f3-df796d231f62-home\" (UniqueName: \"kubernetes.io/empty-dir/c46dca02-6036-46de-b0f3-df796d231f62-home\") pod \"c46dca02-6036-46de-b0f3-df796d231f62\" (UID: \"c46dca02-6036-46de-b0f3-df796d231f62\") " Nov 03 02:40:27.567715 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:27.567335 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c46dca02-6036-46de-b0f3-df796d231f62-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c46dca02-6036-46de-b0f3-df796d231f62-dshm\") pod \"c46dca02-6036-46de-b0f3-df796d231f62\" (UID: \"c46dca02-6036-46de-b0f3-df796d231f62\") " Nov 03 02:40:27.567715 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:27.567377 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a87f809-0b26-480e-bb37-ff234a40c5c5-model-cache" pod "8a87f809-0b26-480e-bb37-ff234a40c5c5" (UID: "8a87f809-0b26-480e-bb37-ff234a40c5c5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:40:27.567715 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:27.567404 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a87f809-0b26-480e-bb37-ff234a40c5c5-kube-api-access-cwqzh\" (UniqueName: \"kubernetes.io/projected/8a87f809-0b26-480e-bb37-ff234a40c5c5-kube-api-access-cwqzh\") pod \"8a87f809-0b26-480e-bb37-ff234a40c5c5\" (UID: \"8a87f809-0b26-480e-bb37-ff234a40c5c5\") " Nov 03 02:40:27.568182 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:27.567817 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a87f809-0b26-480e-bb37-ff234a40c5c5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a87f809-0b26-480e-bb37-ff234a40c5c5-dshm\") pod \"8a87f809-0b26-480e-bb37-ff234a40c5c5\" (UID: \"8a87f809-0b26-480e-bb37-ff234a40c5c5\") " Nov 03 02:40:27.568329 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:27.568225 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c46dca02-6036-46de-b0f3-df796d231f62-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c46dca02-6036-46de-b0f3-df796d231f62-model-cache\") pod \"c46dca02-6036-46de-b0f3-df796d231f62\" (UID: \"c46dca02-6036-46de-b0f3-df796d231f62\") " Nov 03 02:40:27.568329 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:27.568305 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c46dca02-6036-46de-b0f3-df796d231f62-kube-api-access-4sl78\" (UniqueName: \"kubernetes.io/projected/c46dca02-6036-46de-b0f3-df796d231f62-kube-api-access-4sl78\") pod \"c46dca02-6036-46de-b0f3-df796d231f62\" (UID: \"c46dca02-6036-46de-b0f3-df796d231f62\") " Nov 03 02:40:27.568555 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:27.568346 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c46dca02-6036-46de-b0f3-df796d231f62-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c46dca02-6036-46de-b0f3-df796d231f62-tmp-dir\") pod \"c46dca02-6036-46de-b0f3-df796d231f62\" (UID: \"c46dca02-6036-46de-b0f3-df796d231f62\") " Nov 03 02:40:27.568555 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:27.568451 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c46dca02-6036-46de-b0f3-df796d231f62-tls-certs\" (UniqueName: \"kubernetes.io/secret/c46dca02-6036-46de-b0f3-df796d231f62-tls-certs\") pod \"c46dca02-6036-46de-b0f3-df796d231f62\" (UID: \"c46dca02-6036-46de-b0f3-df796d231f62\") " Nov 03 02:40:27.568555 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:27.568487 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a87f809-0b26-480e-bb37-ff234a40c5c5-home\" (UniqueName: \"kubernetes.io/empty-dir/8a87f809-0b26-480e-bb37-ff234a40c5c5-home\") pod \"8a87f809-0b26-480e-bb37-ff234a40c5c5\" (UID: \"8a87f809-0b26-480e-bb37-ff234a40c5c5\") " Nov 03 02:40:27.569108 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:27.569082 5437 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a87f809-0b26-480e-bb37-ff234a40c5c5-model-cache\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:40:27.569254 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:27.567950 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a87f809-0b26-480e-bb37-ff234a40c5c5-tmp-dir" pod "8a87f809-0b26-480e-bb37-ff234a40c5c5" (UID: "8a87f809-0b26-480e-bb37-ff234a40c5c5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:40:27.569254 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:27.567958 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c46dca02-6036-46de-b0f3-df796d231f62-home" pod "c46dca02-6036-46de-b0f3-df796d231f62" (UID: "c46dca02-6036-46de-b0f3-df796d231f62"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:40:27.569254 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:27.569205 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c46dca02-6036-46de-b0f3-df796d231f62-tmp-dir" pod "c46dca02-6036-46de-b0f3-df796d231f62" (UID: "c46dca02-6036-46de-b0f3-df796d231f62"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:40:27.569807 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:27.569380 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a87f809-0b26-480e-bb37-ff234a40c5c5-home" pod "8a87f809-0b26-480e-bb37-ff234a40c5c5" (UID: "8a87f809-0b26-480e-bb37-ff234a40c5c5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:40:27.569807 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:27.569499 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c46dca02-6036-46de-b0f3-df796d231f62-model-cache" pod "c46dca02-6036-46de-b0f3-df796d231f62" (UID: "c46dca02-6036-46de-b0f3-df796d231f62"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:40:27.570235 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:27.570198 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a87f809-0b26-480e-bb37-ff234a40c5c5-kube-api-access-cwqzh" pod "8a87f809-0b26-480e-bb37-ff234a40c5c5" (UID: "8a87f809-0b26-480e-bb37-ff234a40c5c5"). InnerVolumeSpecName "kube-api-access-cwqzh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:40:27.570235 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:27.570195 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a87f809-0b26-480e-bb37-ff234a40c5c5-tls-certs" pod "8a87f809-0b26-480e-bb37-ff234a40c5c5" (UID: "8a87f809-0b26-480e-bb37-ff234a40c5c5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:40:27.571980 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:27.571915 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c46dca02-6036-46de-b0f3-df796d231f62-tls-certs" pod "c46dca02-6036-46de-b0f3-df796d231f62" (UID: "c46dca02-6036-46de-b0f3-df796d231f62"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:40:27.572574 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:27.572538 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a87f809-0b26-480e-bb37-ff234a40c5c5-dshm" pod "8a87f809-0b26-480e-bb37-ff234a40c5c5" (UID: "8a87f809-0b26-480e-bb37-ff234a40c5c5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:40:27.575847 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:27.575796 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c46dca02-6036-46de-b0f3-df796d231f62-dshm" pod "c46dca02-6036-46de-b0f3-df796d231f62" (UID: "c46dca02-6036-46de-b0f3-df796d231f62"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:40:27.575962 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:27.575859 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c46dca02-6036-46de-b0f3-df796d231f62-kube-api-access-4sl78" pod "c46dca02-6036-46de-b0f3-df796d231f62" (UID: "c46dca02-6036-46de-b0f3-df796d231f62"). InnerVolumeSpecName "kube-api-access-4sl78". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:40:27.670306 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:27.670272 5437 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a87f809-0b26-480e-bb37-ff234a40c5c5-tmp-dir\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:40:27.670306 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:27.670304 5437 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a87f809-0b26-480e-bb37-ff234a40c5c5-tls-certs\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:40:27.670306 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:27.670314 5437 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c46dca02-6036-46de-b0f3-df796d231f62-home\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:40:27.670574 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:27.670326 5437 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c46dca02-6036-46de-b0f3-df796d231f62-dshm\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:40:27.670574 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:27.670336 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwqzh\" (UniqueName: \"kubernetes.io/projected/8a87f809-0b26-480e-bb37-ff234a40c5c5-kube-api-access-cwqzh\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:40:27.670574 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:27.670345 5437 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a87f809-0b26-480e-bb37-ff234a40c5c5-dshm\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:40:27.670574 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:27.670353 5437 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c46dca02-6036-46de-b0f3-df796d231f62-model-cache\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:40:27.670574 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:27.670375 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4sl78\" (UniqueName: \"kubernetes.io/projected/c46dca02-6036-46de-b0f3-df796d231f62-kube-api-access-4sl78\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:40:27.670574 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:27.670388 5437 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c46dca02-6036-46de-b0f3-df796d231f62-tmp-dir\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:40:27.670574 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:27.670396 5437 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c46dca02-6036-46de-b0f3-df796d231f62-tls-certs\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:40:27.670574 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:27.670405 5437 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a87f809-0b26-480e-bb37-ff234a40c5c5-home\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:40:27.926536 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:27.926495 5437 generic.go:356] "Generic (PLEG): container finished" podID="8a87f809-0b26-480e-bb37-ff234a40c5c5" containerID="ff1fece57a2d906fc223c5efd375595ffd5ec9aff94298218b650bd18d6aafcb" exitCode=0 Nov 03 02:40:27.927004 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:27.926543 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-41aa5f/simulator-w3-41aa5f-kserve-586cfbcdd4-gsmfk" Nov 03 02:40:27.927004 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:27.926577 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-41aa5f/simulator-w3-41aa5f-kserve-586cfbcdd4-gsmfk" event={"ID":"8a87f809-0b26-480e-bb37-ff234a40c5c5","Type":"ContainerDied","Data":"ff1fece57a2d906fc223c5efd375595ffd5ec9aff94298218b650bd18d6aafcb"} Nov 03 02:40:27.927004 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:27.926625 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-41aa5f/simulator-w3-41aa5f-kserve-586cfbcdd4-gsmfk" event={"ID":"8a87f809-0b26-480e-bb37-ff234a40c5c5","Type":"ContainerDied","Data":"5b1401233b90161766d39ae0c59c80ee2934ae2bf40ceab42423f60dc3a0d8bd"} Nov 03 02:40:27.927004 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:27.926644 5437 scope.go:122] "RemoveContainer" containerID="ff1fece57a2d906fc223c5efd375595ffd5ec9aff94298218b650bd18d6aafcb" Nov 03 02:40:27.928301 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:27.928280 5437 generic.go:356] "Generic (PLEG): container finished" podID="c46dca02-6036-46de-b0f3-df796d231f62" containerID="a4bacbf873e67a41775fb4df36c84815925b00fb28290cd91cbcc237f594249f" exitCode=0 Nov 03 02:40:27.928409 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:27.928315 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-41aa5f/premium-w3-41aa5f-kserve-5fb94fd5fc-gmtkb" Nov 03 02:40:27.928464 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:27.928355 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-41aa5f/premium-w3-41aa5f-kserve-5fb94fd5fc-gmtkb" event={"ID":"c46dca02-6036-46de-b0f3-df796d231f62","Type":"ContainerDied","Data":"a4bacbf873e67a41775fb4df36c84815925b00fb28290cd91cbcc237f594249f"} Nov 03 02:40:27.928517 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:27.928469 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-41aa5f/premium-w3-41aa5f-kserve-5fb94fd5fc-gmtkb" event={"ID":"c46dca02-6036-46de-b0f3-df796d231f62","Type":"ContainerDied","Data":"c59de78b693f8e4a644b8940c602fb78dffbcef6d6d9e4121cc6a1b346f3054e"} Nov 03 02:40:27.946572 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:27.946541 5437 scope.go:122] "RemoveContainer" containerID="ff1fece57a2d906fc223c5efd375595ffd5ec9aff94298218b650bd18d6aafcb" Nov 03 02:40:27.947031 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:40:27.946999 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff1fece57a2d906fc223c5efd375595ffd5ec9aff94298218b650bd18d6aafcb\": container with ID starting with ff1fece57a2d906fc223c5efd375595ffd5ec9aff94298218b650bd18d6aafcb not found: ID does not exist" containerID="ff1fece57a2d906fc223c5efd375595ffd5ec9aff94298218b650bd18d6aafcb" Nov 03 02:40:27.948213 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:27.947042 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff1fece57a2d906fc223c5efd375595ffd5ec9aff94298218b650bd18d6aafcb"} err="failed to get container status \"ff1fece57a2d906fc223c5efd375595ffd5ec9aff94298218b650bd18d6aafcb\": rpc error: code = NotFound desc = could not find container \"ff1fece57a2d906fc223c5efd375595ffd5ec9aff94298218b650bd18d6aafcb\": container with ID starting with ff1fece57a2d906fc223c5efd375595ffd5ec9aff94298218b650bd18d6aafcb not found: ID does not exist" Nov 03 02:40:27.948213 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:27.947070 5437 scope.go:122] "RemoveContainer" containerID="a4bacbf873e67a41775fb4df36c84815925b00fb28290cd91cbcc237f594249f" Nov 03 02:40:27.962916 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:27.962873 5437 scope.go:122] "RemoveContainer" containerID="a4bacbf873e67a41775fb4df36c84815925b00fb28290cd91cbcc237f594249f" Nov 03 02:40:27.963259 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:40:27.963228 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4bacbf873e67a41775fb4df36c84815925b00fb28290cd91cbcc237f594249f\": container with ID starting with a4bacbf873e67a41775fb4df36c84815925b00fb28290cd91cbcc237f594249f not found: ID does not exist" containerID="a4bacbf873e67a41775fb4df36c84815925b00fb28290cd91cbcc237f594249f" Nov 03 02:40:27.963390 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:27.963268 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4bacbf873e67a41775fb4df36c84815925b00fb28290cd91cbcc237f594249f"} err="failed to get container status \"a4bacbf873e67a41775fb4df36c84815925b00fb28290cd91cbcc237f594249f\": rpc error: code = NotFound desc = could not find container \"a4bacbf873e67a41775fb4df36c84815925b00fb28290cd91cbcc237f594249f\": container with ID starting with a4bacbf873e67a41775fb4df36c84815925b00fb28290cd91cbcc237f594249f not found: ID does not exist" Nov 03 02:40:28.096693 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:28.096659 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bcc4bb00-2f9e-4f38-a331-f725729873f0" path="/var/lib/kubelet/pods/bcc4bb00-2f9e-4f38-a331-f725729873f0/volumes" Nov 03 02:40:28.097279 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:28.097254 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-41aa5f/simulator-w3-41aa5f-kserve-586cfbcdd4-gsmfk"] Nov 03 02:40:28.100052 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:28.100019 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-41aa5f/simulator-w3-41aa5f-kserve-586cfbcdd4-gsmfk"] Nov 03 02:40:28.213018 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:28.212930 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-41aa5f/premium-w3-41aa5f-kserve-5fb94fd5fc-gmtkb"] Nov 03 02:40:28.218278 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:28.218248 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-41aa5f/premium-w3-41aa5f-kserve-5fb94fd5fc-gmtkb"] Nov 03 02:40:28.780535 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:28.780483 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:40:29.797560 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:29.797524 5437 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-k42gq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 02:40:29.797946 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:29.797581 5437 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k42gq" podUID="615071b1-f5f9-4d03-90b8-1210c450902d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 02:40:30.089546 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:30.089466 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a87f809-0b26-480e-bb37-ff234a40c5c5" path="/var/lib/kubelet/pods/8a87f809-0b26-480e-bb37-ff234a40c5c5/volumes" Nov 03 02:40:30.089914 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:30.089901 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c46dca02-6036-46de-b0f3-df796d231f62" path="/var/lib/kubelet/pods/c46dca02-6036-46de-b0f3-df796d231f62/volumes" Nov 03 02:40:30.551968 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:30.551931 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:40:32.553445 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:32.553408 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:40:32.755793 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:32.755765 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k42gq" Nov 03 02:40:32.822125 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:32.822091 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:40:32.930100 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:32.930069 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/615071b1-f5f9-4d03-90b8-1210c450902d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/615071b1-f5f9-4d03-90b8-1210c450902d-istiod-ca-cert\") pod \"615071b1-f5f9-4d03-90b8-1210c450902d\" (UID: \"615071b1-f5f9-4d03-90b8-1210c450902d\") " Nov 03 02:40:32.930273 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:32.930126 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/615071b1-f5f9-4d03-90b8-1210c450902d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/615071b1-f5f9-4d03-90b8-1210c450902d-workload-socket\") pod \"615071b1-f5f9-4d03-90b8-1210c450902d\" (UID: \"615071b1-f5f9-4d03-90b8-1210c450902d\") " Nov 03 02:40:32.930273 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:32.930190 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/615071b1-f5f9-4d03-90b8-1210c450902d-istio-token\" (UniqueName: \"kubernetes.io/projected/615071b1-f5f9-4d03-90b8-1210c450902d-istio-token\") pod \"615071b1-f5f9-4d03-90b8-1210c450902d\" (UID: \"615071b1-f5f9-4d03-90b8-1210c450902d\") " Nov 03 02:40:32.930273 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:32.930247 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/615071b1-f5f9-4d03-90b8-1210c450902d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/615071b1-f5f9-4d03-90b8-1210c450902d-istio-data\") pod \"615071b1-f5f9-4d03-90b8-1210c450902d\" (UID: \"615071b1-f5f9-4d03-90b8-1210c450902d\") " Nov 03 02:40:32.930848 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:32.930285 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/615071b1-f5f9-4d03-90b8-1210c450902d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/615071b1-f5f9-4d03-90b8-1210c450902d-istio-envoy\") pod \"615071b1-f5f9-4d03-90b8-1210c450902d\" (UID: \"615071b1-f5f9-4d03-90b8-1210c450902d\") " Nov 03 02:40:32.930848 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:32.930332 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/615071b1-f5f9-4d03-90b8-1210c450902d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/615071b1-f5f9-4d03-90b8-1210c450902d-istio-podinfo\") pod \"615071b1-f5f9-4d03-90b8-1210c450902d\" (UID: \"615071b1-f5f9-4d03-90b8-1210c450902d\") " Nov 03 02:40:32.930848 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:32.930391 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/615071b1-f5f9-4d03-90b8-1210c450902d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/615071b1-f5f9-4d03-90b8-1210c450902d-credential-socket\") pod \"615071b1-f5f9-4d03-90b8-1210c450902d\" (UID: \"615071b1-f5f9-4d03-90b8-1210c450902d\") " Nov 03 02:40:32.930848 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:32.930470 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/615071b1-f5f9-4d03-90b8-1210c450902d-kube-api-access-6cp6n\" (UniqueName: \"kubernetes.io/projected/615071b1-f5f9-4d03-90b8-1210c450902d-kube-api-access-6cp6n\") pod \"615071b1-f5f9-4d03-90b8-1210c450902d\" (UID: \"615071b1-f5f9-4d03-90b8-1210c450902d\") " Nov 03 02:40:32.930848 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:32.930487 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/615071b1-f5f9-4d03-90b8-1210c450902d-istiod-ca-cert" pod "615071b1-f5f9-4d03-90b8-1210c450902d" (UID: "615071b1-f5f9-4d03-90b8-1210c450902d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:40:32.930848 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:32.930521 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/615071b1-f5f9-4d03-90b8-1210c450902d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/615071b1-f5f9-4d03-90b8-1210c450902d-workload-certs\") pod \"615071b1-f5f9-4d03-90b8-1210c450902d\" (UID: \"615071b1-f5f9-4d03-90b8-1210c450902d\") " Nov 03 02:40:32.930848 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:32.930572 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/615071b1-f5f9-4d03-90b8-1210c450902d-workload-socket" pod "615071b1-f5f9-4d03-90b8-1210c450902d" (UID: "615071b1-f5f9-4d03-90b8-1210c450902d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:40:32.931162 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:32.930933 5437 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/615071b1-f5f9-4d03-90b8-1210c450902d-istiod-ca-cert\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:40:32.931162 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:32.930956 5437 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/615071b1-f5f9-4d03-90b8-1210c450902d-workload-socket\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:40:32.931162 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:32.931040 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/615071b1-f5f9-4d03-90b8-1210c450902d-credential-socket" pod "615071b1-f5f9-4d03-90b8-1210c450902d" (UID: "615071b1-f5f9-4d03-90b8-1210c450902d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:40:32.931335 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:32.931242 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/615071b1-f5f9-4d03-90b8-1210c450902d-istio-data" pod "615071b1-f5f9-4d03-90b8-1210c450902d" (UID: "615071b1-f5f9-4d03-90b8-1210c450902d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:40:32.932036 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:32.932013 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/615071b1-f5f9-4d03-90b8-1210c450902d-workload-certs" pod "615071b1-f5f9-4d03-90b8-1210c450902d" (UID: "615071b1-f5f9-4d03-90b8-1210c450902d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:40:32.933170 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:32.933133 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/615071b1-f5f9-4d03-90b8-1210c450902d-istio-token" pod "615071b1-f5f9-4d03-90b8-1210c450902d" (UID: "615071b1-f5f9-4d03-90b8-1210c450902d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:40:32.933590 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:32.933564 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/615071b1-f5f9-4d03-90b8-1210c450902d-istio-envoy" pod "615071b1-f5f9-4d03-90b8-1210c450902d" (UID: "615071b1-f5f9-4d03-90b8-1210c450902d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:40:32.933708 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:32.933679 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/615071b1-f5f9-4d03-90b8-1210c450902d-istio-podinfo" pod "615071b1-f5f9-4d03-90b8-1210c450902d" (UID: "615071b1-f5f9-4d03-90b8-1210c450902d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:40:32.933872 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:32.933851 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/615071b1-f5f9-4d03-90b8-1210c450902d-kube-api-access-6cp6n" pod "615071b1-f5f9-4d03-90b8-1210c450902d" (UID: "615071b1-f5f9-4d03-90b8-1210c450902d"). InnerVolumeSpecName "kube-api-access-6cp6n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:40:32.961023 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:32.960990 5437 generic.go:356] "Generic (PLEG): container finished" podID="615071b1-f5f9-4d03-90b8-1210c450902d" containerID="6049cceacbaaff9610f640c7b31bc02e38872a068dce6da6c2782a5c48be6e06" exitCode=0 Nov 03 02:40:32.961206 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:32.961060 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k42gq" Nov 03 02:40:32.961206 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:32.961072 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k42gq" event={"ID":"615071b1-f5f9-4d03-90b8-1210c450902d","Type":"ContainerDied","Data":"6049cceacbaaff9610f640c7b31bc02e38872a068dce6da6c2782a5c48be6e06"} Nov 03 02:40:32.961206 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:32.961147 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k42gq" event={"ID":"615071b1-f5f9-4d03-90b8-1210c450902d","Type":"ContainerDied","Data":"b3ba2279c7ef03f1832d6c523a37505ec0f41ecab5dfdd1cb4d6de22fabcbafb"} Nov 03 02:40:32.961206 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:32.961178 5437 scope.go:122] "RemoveContainer" containerID="6049cceacbaaff9610f640c7b31bc02e38872a068dce6da6c2782a5c48be6e06" Nov 03 02:40:32.971854 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:32.971830 5437 scope.go:122] "RemoveContainer" containerID="6049cceacbaaff9610f640c7b31bc02e38872a068dce6da6c2782a5c48be6e06" Nov 03 02:40:32.972572 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:40:32.972542 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6049cceacbaaff9610f640c7b31bc02e38872a068dce6da6c2782a5c48be6e06\": container with ID starting with 6049cceacbaaff9610f640c7b31bc02e38872a068dce6da6c2782a5c48be6e06 not found: ID does not exist" containerID="6049cceacbaaff9610f640c7b31bc02e38872a068dce6da6c2782a5c48be6e06" Nov 03 02:40:32.972639 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:32.972584 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6049cceacbaaff9610f640c7b31bc02e38872a068dce6da6c2782a5c48be6e06"} err="failed to get container status \"6049cceacbaaff9610f640c7b31bc02e38872a068dce6da6c2782a5c48be6e06\": rpc error: code = NotFound desc = could not find container \"6049cceacbaaff9610f640c7b31bc02e38872a068dce6da6c2782a5c48be6e06\": container with ID starting with 6049cceacbaaff9610f640c7b31bc02e38872a068dce6da6c2782a5c48be6e06 not found: ID does not exist" Nov 03 02:40:33.031910 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:33.031877 5437 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/615071b1-f5f9-4d03-90b8-1210c450902d-istio-token\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:40:33.031910 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:33.031904 5437 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/615071b1-f5f9-4d03-90b8-1210c450902d-istio-data\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:40:33.031910 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:33.031914 5437 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/615071b1-f5f9-4d03-90b8-1210c450902d-istio-envoy\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:40:33.032136 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:33.031924 5437 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/615071b1-f5f9-4d03-90b8-1210c450902d-istio-podinfo\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:40:33.032136 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:33.031934 5437 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/615071b1-f5f9-4d03-90b8-1210c450902d-credential-socket\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:40:33.032136 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:33.031946 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cp6n\" (UniqueName: \"kubernetes.io/projected/615071b1-f5f9-4d03-90b8-1210c450902d-kube-api-access-6cp6n\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:40:33.032136 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:33.031955 5437 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/615071b1-f5f9-4d03-90b8-1210c450902d-workload-certs\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:40:33.132056 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:33.131966 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k42gq"] Nov 03 02:40:33.136259 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:33.136229 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k42gq"] Nov 03 02:40:33.766505 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:33.766461 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:40:34.088228 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:34.088191 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="615071b1-f5f9-4d03-90b8-1210c450902d" path="/var/lib/kubelet/pods/615071b1-f5f9-4d03-90b8-1210c450902d/volumes" Nov 03 02:40:34.122826 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:34.122790 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:40:34.571066 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:34.571034 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-98a394-openshift-default-7dff487c9f-rwl2t"] Nov 03 02:40:34.571293 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:34.571261 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-98a394-openshift-default-7dff487c9f-rwl2t" podUID="80edf8e3-195e-4ccf-b826-27adbf5504ef" containerName="istio-proxy" containerID="cri-o://dea919d716e0cfeb35ccbd912a36d53367a04691c9eddabf4764795ad7674381" gracePeriod=30 Nov 03 02:40:34.636610 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:34.636514 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:40:34.737897 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:34.737858 5437 patch_prober.go:29] interesting pod/e2e-shared-a-w2-98a394-openshift-default-7dff487c9f-rwl2t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 02:40:34.738072 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:34.737918 5437 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-98a394-openshift-default-7dff487c9f-rwl2t" podUID="80edf8e3-195e-4ccf-b826-27adbf5504ef" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 02:40:35.433485 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:35.433449 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:40:37.275105 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:37.275061 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:40:37.529474 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:37.529348 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:40:38.770025 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:38.769975 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:40:39.046313 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:39.046224 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:40:39.267171 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:39.267136 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:40:39.789744 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:39.789703 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:40:39.845537 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:39.845468 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-98a394-openshift-default-7dff487c9f-rwl2t" Nov 03 02:40:39.899443 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:39.899403 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80edf8e3-195e-4ccf-b826-27adbf5504ef-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80edf8e3-195e-4ccf-b826-27adbf5504ef-credential-socket\") pod \"80edf8e3-195e-4ccf-b826-27adbf5504ef\" (UID: \"80edf8e3-195e-4ccf-b826-27adbf5504ef\") " Nov 03 02:40:39.899633 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:39.899505 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80edf8e3-195e-4ccf-b826-27adbf5504ef-istio-token\" (UniqueName: \"kubernetes.io/projected/80edf8e3-195e-4ccf-b826-27adbf5504ef-istio-token\") pod \"80edf8e3-195e-4ccf-b826-27adbf5504ef\" (UID: \"80edf8e3-195e-4ccf-b826-27adbf5504ef\") " Nov 03 02:40:39.899633 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:39.899548 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80edf8e3-195e-4ccf-b826-27adbf5504ef-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80edf8e3-195e-4ccf-b826-27adbf5504ef-istiod-ca-cert\") pod \"80edf8e3-195e-4ccf-b826-27adbf5504ef\" (UID: \"80edf8e3-195e-4ccf-b826-27adbf5504ef\") " Nov 03 02:40:39.899633 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:39.899623 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80edf8e3-195e-4ccf-b826-27adbf5504ef-kube-api-access-dl6f7\" (UniqueName: \"kubernetes.io/projected/80edf8e3-195e-4ccf-b826-27adbf5504ef-kube-api-access-dl6f7\") pod \"80edf8e3-195e-4ccf-b826-27adbf5504ef\" (UID: \"80edf8e3-195e-4ccf-b826-27adbf5504ef\") " Nov 03 02:40:39.899838 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:39.899660 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80edf8e3-195e-4ccf-b826-27adbf5504ef-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80edf8e3-195e-4ccf-b826-27adbf5504ef-workload-socket\") pod \"80edf8e3-195e-4ccf-b826-27adbf5504ef\" (UID: \"80edf8e3-195e-4ccf-b826-27adbf5504ef\") " Nov 03 02:40:39.899838 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:39.899752 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80edf8e3-195e-4ccf-b826-27adbf5504ef-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80edf8e3-195e-4ccf-b826-27adbf5504ef-istio-data\") pod \"80edf8e3-195e-4ccf-b826-27adbf5504ef\" (UID: \"80edf8e3-195e-4ccf-b826-27adbf5504ef\") " Nov 03 02:40:39.899838 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:39.899796 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80edf8e3-195e-4ccf-b826-27adbf5504ef-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80edf8e3-195e-4ccf-b826-27adbf5504ef-istio-envoy\") pod \"80edf8e3-195e-4ccf-b826-27adbf5504ef\" (UID: \"80edf8e3-195e-4ccf-b826-27adbf5504ef\") " Nov 03 02:40:39.899838 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:39.899804 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80edf8e3-195e-4ccf-b826-27adbf5504ef-credential-socket" pod "80edf8e3-195e-4ccf-b826-27adbf5504ef" (UID: "80edf8e3-195e-4ccf-b826-27adbf5504ef"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:40:39.899838 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:39.899834 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80edf8e3-195e-4ccf-b826-27adbf5504ef-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80edf8e3-195e-4ccf-b826-27adbf5504ef-workload-certs\") pod \"80edf8e3-195e-4ccf-b826-27adbf5504ef\" (UID: \"80edf8e3-195e-4ccf-b826-27adbf5504ef\") " Nov 03 02:40:39.900160 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:39.899871 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/80edf8e3-195e-4ccf-b826-27adbf5504ef-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80edf8e3-195e-4ccf-b826-27adbf5504ef-istio-podinfo\") pod \"80edf8e3-195e-4ccf-b826-27adbf5504ef\" (UID: \"80edf8e3-195e-4ccf-b826-27adbf5504ef\") " Nov 03 02:40:39.900160 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:39.900039 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80edf8e3-195e-4ccf-b826-27adbf5504ef-workload-socket" pod "80edf8e3-195e-4ccf-b826-27adbf5504ef" (UID: "80edf8e3-195e-4ccf-b826-27adbf5504ef"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:40:39.900160 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:39.900096 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80edf8e3-195e-4ccf-b826-27adbf5504ef-istiod-ca-cert" pod "80edf8e3-195e-4ccf-b826-27adbf5504ef" (UID: "80edf8e3-195e-4ccf-b826-27adbf5504ef"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:40:39.900340 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:39.900329 5437 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80edf8e3-195e-4ccf-b826-27adbf5504ef-credential-socket\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:40:39.900441 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:39.900351 5437 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80edf8e3-195e-4ccf-b826-27adbf5504ef-istiod-ca-cert\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:40:39.900441 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:39.900410 5437 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80edf8e3-195e-4ccf-b826-27adbf5504ef-workload-socket\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:40:39.900576 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:39.900548 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80edf8e3-195e-4ccf-b826-27adbf5504ef-workload-certs" pod "80edf8e3-195e-4ccf-b826-27adbf5504ef" (UID: "80edf8e3-195e-4ccf-b826-27adbf5504ef"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:40:39.900783 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:39.900754 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80edf8e3-195e-4ccf-b826-27adbf5504ef-istio-data" pod "80edf8e3-195e-4ccf-b826-27adbf5504ef" (UID: "80edf8e3-195e-4ccf-b826-27adbf5504ef"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:40:39.902170 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:39.902143 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80edf8e3-195e-4ccf-b826-27adbf5504ef-istio-token" pod "80edf8e3-195e-4ccf-b826-27adbf5504ef" (UID: "80edf8e3-195e-4ccf-b826-27adbf5504ef"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:40:39.902273 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:39.902175 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80edf8e3-195e-4ccf-b826-27adbf5504ef-kube-api-access-dl6f7" pod "80edf8e3-195e-4ccf-b826-27adbf5504ef" (UID: "80edf8e3-195e-4ccf-b826-27adbf5504ef"). InnerVolumeSpecName "kube-api-access-dl6f7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:40:39.902273 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:39.902228 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/80edf8e3-195e-4ccf-b826-27adbf5504ef-istio-podinfo" pod "80edf8e3-195e-4ccf-b826-27adbf5504ef" (UID: "80edf8e3-195e-4ccf-b826-27adbf5504ef"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:40:39.902351 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:39.902275 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80edf8e3-195e-4ccf-b826-27adbf5504ef-istio-envoy" pod "80edf8e3-195e-4ccf-b826-27adbf5504ef" (UID: "80edf8e3-195e-4ccf-b826-27adbf5504ef"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:40:40.001078 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:40.001047 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dl6f7\" (UniqueName: \"kubernetes.io/projected/80edf8e3-195e-4ccf-b826-27adbf5504ef-kube-api-access-dl6f7\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:40:40.001078 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:40.001073 5437 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80edf8e3-195e-4ccf-b826-27adbf5504ef-istio-data\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:40:40.001309 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:40.001085 5437 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80edf8e3-195e-4ccf-b826-27adbf5504ef-istio-envoy\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:40:40.001309 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:40.001100 5437 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80edf8e3-195e-4ccf-b826-27adbf5504ef-workload-certs\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:40:40.001309 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:40.001110 5437 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80edf8e3-195e-4ccf-b826-27adbf5504ef-istio-podinfo\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:40:40.001309 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:40.001124 5437 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80edf8e3-195e-4ccf-b826-27adbf5504ef-istio-token\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:40:40.002234 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:40.002206 5437 generic.go:356] "Generic (PLEG): container finished" podID="80edf8e3-195e-4ccf-b826-27adbf5504ef" containerID="dea919d716e0cfeb35ccbd912a36d53367a04691c9eddabf4764795ad7674381" exitCode=0 Nov 03 02:40:40.002392 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:40.002266 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-98a394-openshift-default-7dff487c9f-rwl2t" Nov 03 02:40:40.002392 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:40.002293 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-98a394-openshift-default-7dff487c9f-rwl2t" event={"ID":"80edf8e3-195e-4ccf-b826-27adbf5504ef","Type":"ContainerDied","Data":"dea919d716e0cfeb35ccbd912a36d53367a04691c9eddabf4764795ad7674381"} Nov 03 02:40:40.002392 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:40.002349 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-98a394-openshift-default-7dff487c9f-rwl2t" event={"ID":"80edf8e3-195e-4ccf-b826-27adbf5504ef","Type":"ContainerDied","Data":"e3c9554a9ca4a5c50feb27dc606efdab67bcce385a87a42ab0c19c75ef150639"} Nov 03 02:40:40.002392 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:40.002385 5437 scope.go:122] "RemoveContainer" containerID="dea919d716e0cfeb35ccbd912a36d53367a04691c9eddabf4764795ad7674381" Nov 03 02:40:40.012284 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:40.012266 5437 scope.go:122] "RemoveContainer" containerID="29ac49292258890b95a12c62bf5b229da77fa6281b14f9711a199a784f3f8876" Nov 03 02:40:40.030100 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:40.030071 5437 scope.go:122] "RemoveContainer" containerID="dea919d716e0cfeb35ccbd912a36d53367a04691c9eddabf4764795ad7674381" Nov 03 02:40:40.030569 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:40:40.030540 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dea919d716e0cfeb35ccbd912a36d53367a04691c9eddabf4764795ad7674381\": container with ID starting with dea919d716e0cfeb35ccbd912a36d53367a04691c9eddabf4764795ad7674381 not found: ID does not exist" containerID="dea919d716e0cfeb35ccbd912a36d53367a04691c9eddabf4764795ad7674381" Nov 03 02:40:40.030678 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:40.030583 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dea919d716e0cfeb35ccbd912a36d53367a04691c9eddabf4764795ad7674381"} err="failed to get container status \"dea919d716e0cfeb35ccbd912a36d53367a04691c9eddabf4764795ad7674381\": rpc error: code = NotFound desc = could not find container \"dea919d716e0cfeb35ccbd912a36d53367a04691c9eddabf4764795ad7674381\": container with ID starting with dea919d716e0cfeb35ccbd912a36d53367a04691c9eddabf4764795ad7674381 not found: ID does not exist" Nov 03 02:40:40.030678 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:40.030615 5437 scope.go:122] "RemoveContainer" containerID="29ac49292258890b95a12c62bf5b229da77fa6281b14f9711a199a784f3f8876" Nov 03 02:40:40.030962 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:40:40.030939 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29ac49292258890b95a12c62bf5b229da77fa6281b14f9711a199a784f3f8876\": container with ID starting with 29ac49292258890b95a12c62bf5b229da77fa6281b14f9711a199a784f3f8876 not found: ID does not exist" containerID="29ac49292258890b95a12c62bf5b229da77fa6281b14f9711a199a784f3f8876" Nov 03 02:40:40.031044 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:40.030971 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29ac49292258890b95a12c62bf5b229da77fa6281b14f9711a199a784f3f8876"} err="failed to get container status \"29ac49292258890b95a12c62bf5b229da77fa6281b14f9711a199a784f3f8876\": rpc error: code = NotFound desc = could not find container \"29ac49292258890b95a12c62bf5b229da77fa6281b14f9711a199a784f3f8876\": container with ID starting with 29ac49292258890b95a12c62bf5b229da77fa6281b14f9711a199a784f3f8876 not found: ID does not exist" Nov 03 02:40:40.144004 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:40.143899 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-98a394-openshift-default-7dff487c9f-rwl2t"] Nov 03 02:40:40.167337 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:40.167295 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-98a394-openshift-default-7dff487c9f-rwl2t"] Nov 03 02:40:42.088140 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:42.088109 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80edf8e3-195e-4ccf-b826-27adbf5504ef" path="/var/lib/kubelet/pods/80edf8e3-195e-4ccf-b826-27adbf5504ef/volumes" Nov 03 02:40:42.500775 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:42.500674 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-6a5d73-695fb45cff-zbh87"] Nov 03 02:40:42.583947 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:42.583898 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:40:42.604562 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:42.604528 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6a5d73-78cd98899-wxsl9"] Nov 03 02:40:42.901867 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:42.901822 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:40:43.103661 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:43.103622 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:40:43.912973 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:43.912934 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:40:44.036842 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:40:44.036803 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 02:40:44.665301 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:44.665263 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-0bf34a/simulator-w4-0bf34a-kserve-6bc96559d9-gkdwl"] Nov 03 02:40:44.863437 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:44.863404 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:40:46.965053 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:46.965011 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:40:47.605112 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:47.605067 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6a5d73-78cd98899-wxsl9" podUID="b5d7ce00-728b-437c-9fbd-524d8c4697fc" containerName="payload-pre-processing" containerID="cri-o://776902be7b8953f6468b3a73028beb618ac8ad691c974b45fb6c5bb49015b0a4" gracePeriod=25 Nov 03 02:40:48.019424 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:48.019304 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6a5d73-695fb45cff-zbh87" podUID="8374d59d-7bed-4b5d-b887-013569604481" containerName="payload-processing" containerID="cri-o://7286af09f0fb4080875634c63936c707d51a488fc6d9033746e21c69574eb205" gracePeriod=25 Nov 03 02:40:48.503670 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:48.503633 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:40:49.066150 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:49.066116 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:40:49.267456 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:49.267420 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:40:52.864472 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:52.864431 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:40:53.693505 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:53.693468 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:40:54.076092 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:54.076055 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:40:54.399669 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:54.399576 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-0bf34a-66c455cb47-xbxp2"] Nov 03 02:40:54.705993 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:54.705791 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:40:54.910887 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:54.910845 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:40:56.300608 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:56.300561 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:40:56.518129 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:56.518093 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:40:56.821113 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:56.821072 5437 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6a5d73-78cd98899-wxsl9" podUID="b5d7ce00-728b-437c-9fbd-524d8c4697fc" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 02:40:56.821297 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:56.821084 5437 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6a5d73-695fb45cff-zbh87" podUID="8374d59d-7bed-4b5d-b887-013569604481" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 02:40:59.099111 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:59.099071 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:40:59.399328 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:59.399226 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:40:59.400047 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:59.400021 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0bf34a-66c455cb47-xbxp2" podUID="28d9cb53-e595-4660-9656-0ef00e30d0d4" containerName="payload-pre-processing" containerID="cri-o://1131d4eaeeb8df52191b82138c24643d48e13b6b3dab9cc1546a4675639868ba" gracePeriod=25 Nov 03 02:40:59.686948 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:59.686850 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-0bf34a/simulator-w4-0bf34a-kserve-6bc96559d9-gkdwl" podUID="4cfca4a1-0da6-4637-bc48-8e65776b9ca9" containerName="main" containerID="cri-o://e538f12b29b5f83223c5f12bec4ba1f77361526abf061af9bdbf15490c810688" gracePeriod=45 Nov 03 02:40:59.936296 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:40:59.936267 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-0bf34a/simulator-w4-0bf34a-kserve-6bc96559d9-gkdwl" Nov 03 02:41:00.024294 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:00.024210 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4cfca4a1-0da6-4637-bc48-8e65776b9ca9-tls-certs\" (UniqueName: \"kubernetes.io/secret/4cfca4a1-0da6-4637-bc48-8e65776b9ca9-tls-certs\") pod \"4cfca4a1-0da6-4637-bc48-8e65776b9ca9\" (UID: \"4cfca4a1-0da6-4637-bc48-8e65776b9ca9\") " Nov 03 02:41:00.024294 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:00.024254 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cfca4a1-0da6-4637-bc48-8e65776b9ca9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4cfca4a1-0da6-4637-bc48-8e65776b9ca9-model-cache\") pod \"4cfca4a1-0da6-4637-bc48-8e65776b9ca9\" (UID: \"4cfca4a1-0da6-4637-bc48-8e65776b9ca9\") " Nov 03 02:41:00.024566 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:00.024417 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cfca4a1-0da6-4637-bc48-8e65776b9ca9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4cfca4a1-0da6-4637-bc48-8e65776b9ca9-dshm\") pod \"4cfca4a1-0da6-4637-bc48-8e65776b9ca9\" (UID: \"4cfca4a1-0da6-4637-bc48-8e65776b9ca9\") " Nov 03 02:41:00.024566 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:00.024476 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cfca4a1-0da6-4637-bc48-8e65776b9ca9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cfca4a1-0da6-4637-bc48-8e65776b9ca9-tmp-dir\") pod \"4cfca4a1-0da6-4637-bc48-8e65776b9ca9\" (UID: \"4cfca4a1-0da6-4637-bc48-8e65776b9ca9\") " Nov 03 02:41:00.024566 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:00.024546 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cfca4a1-0da6-4637-bc48-8e65776b9ca9-model-cache" pod "4cfca4a1-0da6-4637-bc48-8e65776b9ca9" (UID: "4cfca4a1-0da6-4637-bc48-8e65776b9ca9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:41:00.024748 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:00.024568 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cfca4a1-0da6-4637-bc48-8e65776b9ca9-kube-api-access-prtvj\" (UniqueName: \"kubernetes.io/projected/4cfca4a1-0da6-4637-bc48-8e65776b9ca9-kube-api-access-prtvj\") pod \"4cfca4a1-0da6-4637-bc48-8e65776b9ca9\" (UID: \"4cfca4a1-0da6-4637-bc48-8e65776b9ca9\") " Nov 03 02:41:00.024748 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:00.024595 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cfca4a1-0da6-4637-bc48-8e65776b9ca9-home\" (UniqueName: \"kubernetes.io/empty-dir/4cfca4a1-0da6-4637-bc48-8e65776b9ca9-home\") pod \"4cfca4a1-0da6-4637-bc48-8e65776b9ca9\" (UID: \"4cfca4a1-0da6-4637-bc48-8e65776b9ca9\") " Nov 03 02:41:00.024852 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:00.024757 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cfca4a1-0da6-4637-bc48-8e65776b9ca9-tmp-dir" pod "4cfca4a1-0da6-4637-bc48-8e65776b9ca9" (UID: "4cfca4a1-0da6-4637-bc48-8e65776b9ca9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:41:00.024908 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:00.024889 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cfca4a1-0da6-4637-bc48-8e65776b9ca9-home" pod "4cfca4a1-0da6-4637-bc48-8e65776b9ca9" (UID: "4cfca4a1-0da6-4637-bc48-8e65776b9ca9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:41:00.025083 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:00.025066 5437 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4cfca4a1-0da6-4637-bc48-8e65776b9ca9-model-cache\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:41:00.025165 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:00.025084 5437 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cfca4a1-0da6-4637-bc48-8e65776b9ca9-tmp-dir\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:41:00.025165 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:00.025093 5437 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4cfca4a1-0da6-4637-bc48-8e65776b9ca9-home\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:41:00.026611 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:00.026583 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4cfca4a1-0da6-4637-bc48-8e65776b9ca9-tls-certs" pod "4cfca4a1-0da6-4637-bc48-8e65776b9ca9" (UID: "4cfca4a1-0da6-4637-bc48-8e65776b9ca9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:41:00.026740 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:00.026690 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cfca4a1-0da6-4637-bc48-8e65776b9ca9-dshm" pod "4cfca4a1-0da6-4637-bc48-8e65776b9ca9" (UID: "4cfca4a1-0da6-4637-bc48-8e65776b9ca9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:41:00.026740 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:00.026715 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cfca4a1-0da6-4637-bc48-8e65776b9ca9-kube-api-access-prtvj" pod "4cfca4a1-0da6-4637-bc48-8e65776b9ca9" (UID: "4cfca4a1-0da6-4637-bc48-8e65776b9ca9"). InnerVolumeSpecName "kube-api-access-prtvj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:41:00.125707 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:00.125666 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prtvj\" (UniqueName: \"kubernetes.io/projected/4cfca4a1-0da6-4637-bc48-8e65776b9ca9-kube-api-access-prtvj\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:41:00.125707 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:00.125702 5437 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4cfca4a1-0da6-4637-bc48-8e65776b9ca9-tls-certs\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:41:00.126216 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:00.125717 5437 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4cfca4a1-0da6-4637-bc48-8e65776b9ca9-dshm\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:41:00.126216 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:00.126074 5437 generic.go:356] "Generic (PLEG): container finished" podID="4cfca4a1-0da6-4637-bc48-8e65776b9ca9" containerID="e538f12b29b5f83223c5f12bec4ba1f77361526abf061af9bdbf15490c810688" exitCode=0 Nov 03 02:41:00.126216 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:00.126123 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-0bf34a/simulator-w4-0bf34a-kserve-6bc96559d9-gkdwl" Nov 03 02:41:00.126216 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:00.126164 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-0bf34a/simulator-w4-0bf34a-kserve-6bc96559d9-gkdwl" event={"ID":"4cfca4a1-0da6-4637-bc48-8e65776b9ca9","Type":"ContainerDied","Data":"e538f12b29b5f83223c5f12bec4ba1f77361526abf061af9bdbf15490c810688"} Nov 03 02:41:00.126415 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:00.126219 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-0bf34a/simulator-w4-0bf34a-kserve-6bc96559d9-gkdwl" event={"ID":"4cfca4a1-0da6-4637-bc48-8e65776b9ca9","Type":"ContainerDied","Data":"4da70f4d20613b0de5e024066ec6c7f16740d10e46eaaae0dfc5a479e777ef5d"} Nov 03 02:41:00.126415 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:00.126245 5437 scope.go:122] "RemoveContainer" containerID="e538f12b29b5f83223c5f12bec4ba1f77361526abf061af9bdbf15490c810688" Nov 03 02:41:00.136956 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:00.136929 5437 scope.go:122] "RemoveContainer" containerID="e538f12b29b5f83223c5f12bec4ba1f77361526abf061af9bdbf15490c810688" Nov 03 02:41:00.137323 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:41:00.137284 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e538f12b29b5f83223c5f12bec4ba1f77361526abf061af9bdbf15490c810688\": container with ID starting with e538f12b29b5f83223c5f12bec4ba1f77361526abf061af9bdbf15490c810688 not found: ID does not exist" containerID="e538f12b29b5f83223c5f12bec4ba1f77361526abf061af9bdbf15490c810688" Nov 03 02:41:00.137445 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:00.137335 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e538f12b29b5f83223c5f12bec4ba1f77361526abf061af9bdbf15490c810688"} err="failed to get container status \"e538f12b29b5f83223c5f12bec4ba1f77361526abf061af9bdbf15490c810688\": rpc error: code = NotFound desc = could not find container \"e538f12b29b5f83223c5f12bec4ba1f77361526abf061af9bdbf15490c810688\": container with ID starting with e538f12b29b5f83223c5f12bec4ba1f77361526abf061af9bdbf15490c810688 not found: ID does not exist" Nov 03 02:41:00.296308 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:00.296231 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-0bf34a/simulator-w4-0bf34a-kserve-6bc96559d9-gkdwl"] Nov 03 02:41:00.301417 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:00.301390 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-0bf34a/simulator-w4-0bf34a-kserve-6bc96559d9-gkdwl"] Nov 03 02:41:01.482008 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:01.481963 5437 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0bf34a-66c455cb47-xbxp2" podUID="28d9cb53-e595-4660-9656-0ef00e30d0d4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 02:41:02.088575 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:02.088539 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4cfca4a1-0da6-4637-bc48-8e65776b9ca9" path="/var/lib/kubelet/pods/4cfca4a1-0da6-4637-bc48-8e65776b9ca9/volumes" Nov 03 02:41:03.496513 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:03.496473 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:41:03.953799 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:03.953765 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:41:04.437971 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:04.437934 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-0bf34a-754fc9fc-tx5ff"] Nov 03 02:41:04.438217 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:04.438191 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0bf34a-754fc9fc-tx5ff" podUID="1f9f6115-560b-451b-ba0d-e66e67ca55eb" containerName="maas-api" containerID="cri-o://4b7fe5d5bc82cb63ab2aa2a35a6c55ccbe003c78663ae91af039c012c6c1c6cf" gracePeriod=30 Nov 03 02:41:04.586698 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:04.586662 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:41:04.696922 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:04.696857 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0bf34a-754fc9fc-tx5ff" Nov 03 02:41:04.878468 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:04.878435 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f9f6115-560b-451b-ba0d-e66e67ca55eb-kube-api-access-cngdj\" (UniqueName: \"kubernetes.io/projected/1f9f6115-560b-451b-ba0d-e66e67ca55eb-kube-api-access-cngdj\") pod \"1f9f6115-560b-451b-ba0d-e66e67ca55eb\" (UID: \"1f9f6115-560b-451b-ba0d-e66e67ca55eb\") " Nov 03 02:41:04.878692 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:04.878513 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f9f6115-560b-451b-ba0d-e66e67ca55eb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f9f6115-560b-451b-ba0d-e66e67ca55eb-metrics-certs\") pod \"1f9f6115-560b-451b-ba0d-e66e67ca55eb\" (UID: \"1f9f6115-560b-451b-ba0d-e66e67ca55eb\") " Nov 03 02:41:04.878692 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:04.878558 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f9f6115-560b-451b-ba0d-e66e67ca55eb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f9f6115-560b-451b-ba0d-e66e67ca55eb-openshift-service-ca\") pod \"1f9f6115-560b-451b-ba0d-e66e67ca55eb\" (UID: \"1f9f6115-560b-451b-ba0d-e66e67ca55eb\") " Nov 03 02:41:04.878692 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:04.878606 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f9f6115-560b-451b-ba0d-e66e67ca55eb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f9f6115-560b-451b-ba0d-e66e67ca55eb-maas-api-tls\") pod \"1f9f6115-560b-451b-ba0d-e66e67ca55eb\" (UID: \"1f9f6115-560b-451b-ba0d-e66e67ca55eb\") " Nov 03 02:41:04.879016 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:04.878985 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f9f6115-560b-451b-ba0d-e66e67ca55eb-openshift-service-ca" pod "1f9f6115-560b-451b-ba0d-e66e67ca55eb" (UID: "1f9f6115-560b-451b-ba0d-e66e67ca55eb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:41:04.880692 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:04.880658 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f9f6115-560b-451b-ba0d-e66e67ca55eb-metrics-certs" pod "1f9f6115-560b-451b-ba0d-e66e67ca55eb" (UID: "1f9f6115-560b-451b-ba0d-e66e67ca55eb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:41:04.880692 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:04.880672 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f9f6115-560b-451b-ba0d-e66e67ca55eb-maas-api-tls" pod "1f9f6115-560b-451b-ba0d-e66e67ca55eb" (UID: "1f9f6115-560b-451b-ba0d-e66e67ca55eb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:41:04.880844 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:04.880752 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f9f6115-560b-451b-ba0d-e66e67ca55eb-kube-api-access-cngdj" pod "1f9f6115-560b-451b-ba0d-e66e67ca55eb" (UID: "1f9f6115-560b-451b-ba0d-e66e67ca55eb"). InnerVolumeSpecName "kube-api-access-cngdj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:41:04.980013 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:04.979925 5437 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f9f6115-560b-451b-ba0d-e66e67ca55eb-openshift-service-ca\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:41:04.980013 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:04.979963 5437 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f9f6115-560b-451b-ba0d-e66e67ca55eb-maas-api-tls\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:41:04.980013 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:04.979973 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cngdj\" (UniqueName: \"kubernetes.io/projected/1f9f6115-560b-451b-ba0d-e66e67ca55eb-kube-api-access-cngdj\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:41:04.980013 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:04.979983 5437 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f9f6115-560b-451b-ba0d-e66e67ca55eb-metrics-certs\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:41:05.045714 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:05.045672 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:41:05.153617 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:05.153577 5437 generic.go:356] "Generic (PLEG): container finished" podID="1f9f6115-560b-451b-ba0d-e66e67ca55eb" containerID="4b7fe5d5bc82cb63ab2aa2a35a6c55ccbe003c78663ae91af039c012c6c1c6cf" exitCode=0 Nov 03 02:41:05.153827 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:05.153626 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0bf34a-754fc9fc-tx5ff" Nov 03 02:41:05.153827 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:05.153655 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0bf34a-754fc9fc-tx5ff" event={"ID":"1f9f6115-560b-451b-ba0d-e66e67ca55eb","Type":"ContainerDied","Data":"4b7fe5d5bc82cb63ab2aa2a35a6c55ccbe003c78663ae91af039c012c6c1c6cf"} Nov 03 02:41:05.153827 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:05.153714 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0bf34a-754fc9fc-tx5ff" event={"ID":"1f9f6115-560b-451b-ba0d-e66e67ca55eb","Type":"ContainerDied","Data":"374eeea253bb92017b76b742e39f7b31e4447cbb789266c6f77f43c7158f115e"} Nov 03 02:41:05.153827 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:05.153737 5437 scope.go:122] "RemoveContainer" containerID="4b7fe5d5bc82cb63ab2aa2a35a6c55ccbe003c78663ae91af039c012c6c1c6cf" Nov 03 02:41:05.164472 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:05.164449 5437 scope.go:122] "RemoveContainer" containerID="4b7fe5d5bc82cb63ab2aa2a35a6c55ccbe003c78663ae91af039c012c6c1c6cf" Nov 03 02:41:05.164788 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:41:05.164764 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b7fe5d5bc82cb63ab2aa2a35a6c55ccbe003c78663ae91af039c012c6c1c6cf\": container with ID starting with 4b7fe5d5bc82cb63ab2aa2a35a6c55ccbe003c78663ae91af039c012c6c1c6cf not found: ID does not exist" containerID="4b7fe5d5bc82cb63ab2aa2a35a6c55ccbe003c78663ae91af039c012c6c1c6cf" Nov 03 02:41:05.164888 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:05.164795 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b7fe5d5bc82cb63ab2aa2a35a6c55ccbe003c78663ae91af039c012c6c1c6cf"} err="failed to get container status \"4b7fe5d5bc82cb63ab2aa2a35a6c55ccbe003c78663ae91af039c012c6c1c6cf\": rpc error: code = NotFound desc = could not find container \"4b7fe5d5bc82cb63ab2aa2a35a6c55ccbe003c78663ae91af039c012c6c1c6cf\": container with ID starting with 4b7fe5d5bc82cb63ab2aa2a35a6c55ccbe003c78663ae91af039c012c6c1c6cf not found: ID does not exist" Nov 03 02:41:05.300851 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:05.300814 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:41:05.324626 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:05.324581 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-0bf34a-754fc9fc-tx5ff"] Nov 03 02:41:05.328041 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:05.328011 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-0bf34a-754fc9fc-tx5ff"] Nov 03 02:41:06.095655 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:06.095614 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f9f6115-560b-451b-ba0d-e66e67ca55eb" path="/var/lib/kubelet/pods/1f9f6115-560b-451b-ba0d-e66e67ca55eb/volumes" Nov 03 02:41:06.288958 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:06.288909 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:41:06.821055 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:06.821011 5437 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6a5d73-695fb45cff-zbh87" podUID="8374d59d-7bed-4b5d-b887-013569604481" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 02:41:06.821220 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:06.821071 5437 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6a5d73-78cd98899-wxsl9" podUID="b5d7ce00-728b-437c-9fbd-524d8c4697fc" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 02:41:07.865531 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:07.865471 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6a5d73-78cd98899-wxsl9" Nov 03 02:41:07.906094 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:07.906065 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5d7ce00-728b-437c-9fbd-524d8c4697fc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5d7ce00-728b-437c-9fbd-524d8c4697fc-plugins-config-volume\") pod \"b5d7ce00-728b-437c-9fbd-524d8c4697fc\" (UID: \"b5d7ce00-728b-437c-9fbd-524d8c4697fc\") " Nov 03 02:41:07.906248 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:07.906160 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5d7ce00-728b-437c-9fbd-524d8c4697fc-kube-api-access-mzztm\" (UniqueName: \"kubernetes.io/projected/b5d7ce00-728b-437c-9fbd-524d8c4697fc-kube-api-access-mzztm\") pod \"b5d7ce00-728b-437c-9fbd-524d8c4697fc\" (UID: \"b5d7ce00-728b-437c-9fbd-524d8c4697fc\") " Nov 03 02:41:07.906455 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:07.906431 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5d7ce00-728b-437c-9fbd-524d8c4697fc-plugins-config-volume" pod "b5d7ce00-728b-437c-9fbd-524d8c4697fc" (UID: "b5d7ce00-728b-437c-9fbd-524d8c4697fc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:41:07.906560 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:07.906536 5437 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5d7ce00-728b-437c-9fbd-524d8c4697fc-plugins-config-volume\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:41:07.908265 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:07.908242 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5d7ce00-728b-437c-9fbd-524d8c4697fc-kube-api-access-mzztm" pod "b5d7ce00-728b-437c-9fbd-524d8c4697fc" (UID: "b5d7ce00-728b-437c-9fbd-524d8c4697fc"). InnerVolumeSpecName "kube-api-access-mzztm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:41:08.008017 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:08.007980 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzztm\" (UniqueName: \"kubernetes.io/projected/b5d7ce00-728b-437c-9fbd-524d8c4697fc-kube-api-access-mzztm\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:41:08.173216 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:08.173182 5437 generic.go:356] "Generic (PLEG): container finished" podID="8374d59d-7bed-4b5d-b887-013569604481" containerID="7286af09f0fb4080875634c63936c707d51a488fc6d9033746e21c69574eb205" exitCode=0 Nov 03 02:41:08.173425 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:08.173252 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6a5d73-695fb45cff-zbh87" event={"ID":"8374d59d-7bed-4b5d-b887-013569604481","Type":"ContainerDied","Data":"7286af09f0fb4080875634c63936c707d51a488fc6d9033746e21c69574eb205"} Nov 03 02:41:08.174724 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:08.174702 5437 generic.go:356] "Generic (PLEG): container finished" podID="b5d7ce00-728b-437c-9fbd-524d8c4697fc" containerID="776902be7b8953f6468b3a73028beb618ac8ad691c974b45fb6c5bb49015b0a4" exitCode=0 Nov 03 02:41:08.174864 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:08.174747 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6a5d73-78cd98899-wxsl9" Nov 03 02:41:08.174864 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:08.174752 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6a5d73-78cd98899-wxsl9" event={"ID":"b5d7ce00-728b-437c-9fbd-524d8c4697fc","Type":"ContainerDied","Data":"776902be7b8953f6468b3a73028beb618ac8ad691c974b45fb6c5bb49015b0a4"} Nov 03 02:41:08.174864 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:08.174847 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6a5d73-78cd98899-wxsl9" event={"ID":"b5d7ce00-728b-437c-9fbd-524d8c4697fc","Type":"ContainerDied","Data":"fcce1a979f7864bca6efa1cd6210dba55a5c5efe387b4d8cd9357fa80544dd6d"} Nov 03 02:41:08.175064 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:08.174871 5437 scope.go:122] "RemoveContainer" containerID="776902be7b8953f6468b3a73028beb618ac8ad691c974b45fb6c5bb49015b0a4" Nov 03 02:41:08.184645 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:08.184618 5437 scope.go:122] "RemoveContainer" containerID="776902be7b8953f6468b3a73028beb618ac8ad691c974b45fb6c5bb49015b0a4" Nov 03 02:41:08.184998 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:41:08.184979 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"776902be7b8953f6468b3a73028beb618ac8ad691c974b45fb6c5bb49015b0a4\": container with ID starting with 776902be7b8953f6468b3a73028beb618ac8ad691c974b45fb6c5bb49015b0a4 not found: ID does not exist" containerID="776902be7b8953f6468b3a73028beb618ac8ad691c974b45fb6c5bb49015b0a4" Nov 03 02:41:08.185072 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:08.185008 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"776902be7b8953f6468b3a73028beb618ac8ad691c974b45fb6c5bb49015b0a4"} err="failed to get container status \"776902be7b8953f6468b3a73028beb618ac8ad691c974b45fb6c5bb49015b0a4\": rpc error: code = NotFound desc = could not find container \"776902be7b8953f6468b3a73028beb618ac8ad691c974b45fb6c5bb49015b0a4\": container with ID starting with 776902be7b8953f6468b3a73028beb618ac8ad691c974b45fb6c5bb49015b0a4 not found: ID does not exist" Nov 03 02:41:08.250077 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:08.250051 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6a5d73-695fb45cff-zbh87" Nov 03 02:41:08.312260 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:08.312221 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8374d59d-7bed-4b5d-b887-013569604481-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8374d59d-7bed-4b5d-b887-013569604481-plugins-config-volume\") pod \"8374d59d-7bed-4b5d-b887-013569604481\" (UID: \"8374d59d-7bed-4b5d-b887-013569604481\") " Nov 03 02:41:08.312443 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:08.312272 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8374d59d-7bed-4b5d-b887-013569604481-kube-api-access-8vsmc\" (UniqueName: \"kubernetes.io/projected/8374d59d-7bed-4b5d-b887-013569604481-kube-api-access-8vsmc\") pod \"8374d59d-7bed-4b5d-b887-013569604481\" (UID: \"8374d59d-7bed-4b5d-b887-013569604481\") " Nov 03 02:41:08.312657 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:08.312636 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8374d59d-7bed-4b5d-b887-013569604481-plugins-config-volume" pod "8374d59d-7bed-4b5d-b887-013569604481" (UID: "8374d59d-7bed-4b5d-b887-013569604481"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:41:08.314354 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:08.314334 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8374d59d-7bed-4b5d-b887-013569604481-kube-api-access-8vsmc" pod "8374d59d-7bed-4b5d-b887-013569604481" (UID: "8374d59d-7bed-4b5d-b887-013569604481"). InnerVolumeSpecName "kube-api-access-8vsmc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:41:08.349626 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:08.349595 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6a5d73-78cd98899-wxsl9"] Nov 03 02:41:08.356562 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:08.356536 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6a5d73-78cd98899-wxsl9"] Nov 03 02:41:08.414100 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:08.414011 5437 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8374d59d-7bed-4b5d-b887-013569604481-plugins-config-volume\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:41:08.414100 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:08.414050 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vsmc\" (UniqueName: \"kubernetes.io/projected/8374d59d-7bed-4b5d-b887-013569604481-kube-api-access-8vsmc\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:41:08.913101 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:08.913061 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:41:09.181771 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:09.181690 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6a5d73-695fb45cff-zbh87" Nov 03 02:41:09.181947 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:09.181696 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6a5d73-695fb45cff-zbh87" event={"ID":"8374d59d-7bed-4b5d-b887-013569604481","Type":"ContainerDied","Data":"43e642965740ec4b1c61d4e3df32c479d4411cd6321fb38804adc3ce95055033"} Nov 03 02:41:09.181947 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:09.181832 5437 scope.go:122] "RemoveContainer" containerID="7286af09f0fb4080875634c63936c707d51a488fc6d9033746e21c69574eb205" Nov 03 02:41:09.354055 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:09.354019 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-6a5d73-695fb45cff-zbh87"] Nov 03 02:41:09.405648 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:09.405617 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-6a5d73-695fb45cff-zbh87"] Nov 03 02:41:10.087307 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:10.087269 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8374d59d-7bed-4b5d-b887-013569604481" path="/var/lib/kubelet/pods/8374d59d-7bed-4b5d-b887-013569604481/volumes" Nov 03 02:41:10.087765 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:10.087747 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5d7ce00-728b-437c-9fbd-524d8c4697fc" path="/var/lib/kubelet/pods/b5d7ce00-728b-437c-9fbd-524d8c4697fc/volumes" Nov 03 02:41:10.588938 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:10.588899 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:41:11.481490 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:11.481449 5437 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0bf34a-66c455cb47-xbxp2" podUID="28d9cb53-e595-4660-9656-0ef00e30d0d4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 02:41:12.275680 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:12.275641 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:41:13.823987 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:13.823949 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:41:14.034675 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:41:14.034633 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 02:41:14.045804 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:14.045773 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:41:14.868527 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:14.868486 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:41:15.072078 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:15.072037 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:41:15.393935 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:15.393877 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:41:15.598734 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:15.598694 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:41:15.862494 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:15.862461 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:41:16.341563 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:16.341524 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:41:16.700561 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:16.700476 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:41:17.151277 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:17.151237 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-wgtp9"] Nov 03 02:41:17.151511 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:17.151486 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-wgtp9" podUID="6807c987-7fd9-4b0d-bee4-dab5799ece82" containerName="istio-proxy" containerID="cri-o://421af643089b5b8096e4e1f8fbac7c9e1f8409c5ef2bf3134677aaec3660fc8d" gracePeriod=30 Nov 03 02:41:17.216660 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:17.216622 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:41:17.421550 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:17.421459 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:41:19.659612 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:19.659550 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0bf34a-66c455cb47-xbxp2" Nov 03 02:41:19.736817 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:19.736780 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28d9cb53-e595-4660-9656-0ef00e30d0d4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28d9cb53-e595-4660-9656-0ef00e30d0d4-plugins-config-volume\") pod \"28d9cb53-e595-4660-9656-0ef00e30d0d4\" (UID: \"28d9cb53-e595-4660-9656-0ef00e30d0d4\") " Nov 03 02:41:19.736993 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:19.736824 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28d9cb53-e595-4660-9656-0ef00e30d0d4-kube-api-access-hfcm8\" (UniqueName: \"kubernetes.io/projected/28d9cb53-e595-4660-9656-0ef00e30d0d4-kube-api-access-hfcm8\") pod \"28d9cb53-e595-4660-9656-0ef00e30d0d4\" (UID: \"28d9cb53-e595-4660-9656-0ef00e30d0d4\") " Nov 03 02:41:19.737256 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:19.737219 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28d9cb53-e595-4660-9656-0ef00e30d0d4-plugins-config-volume" pod "28d9cb53-e595-4660-9656-0ef00e30d0d4" (UID: "28d9cb53-e595-4660-9656-0ef00e30d0d4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:41:19.738946 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:19.738926 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28d9cb53-e595-4660-9656-0ef00e30d0d4-kube-api-access-hfcm8" pod "28d9cb53-e595-4660-9656-0ef00e30d0d4" (UID: "28d9cb53-e595-4660-9656-0ef00e30d0d4"). InnerVolumeSpecName "kube-api-access-hfcm8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:41:19.838448 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:19.838405 5437 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28d9cb53-e595-4660-9656-0ef00e30d0d4-plugins-config-volume\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:41:19.838448 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:19.838439 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfcm8\" (UniqueName: \"kubernetes.io/projected/28d9cb53-e595-4660-9656-0ef00e30d0d4-kube-api-access-hfcm8\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:41:20.244310 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:20.244271 5437 generic.go:356] "Generic (PLEG): container finished" podID="28d9cb53-e595-4660-9656-0ef00e30d0d4" containerID="1131d4eaeeb8df52191b82138c24643d48e13b6b3dab9cc1546a4675639868ba" exitCode=0 Nov 03 02:41:20.244514 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:20.244319 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0bf34a-66c455cb47-xbxp2" Nov 03 02:41:20.244514 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:20.244359 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0bf34a-66c455cb47-xbxp2" event={"ID":"28d9cb53-e595-4660-9656-0ef00e30d0d4","Type":"ContainerDied","Data":"1131d4eaeeb8df52191b82138c24643d48e13b6b3dab9cc1546a4675639868ba"} Nov 03 02:41:20.244514 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:20.244429 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0bf34a-66c455cb47-xbxp2" event={"ID":"28d9cb53-e595-4660-9656-0ef00e30d0d4","Type":"ContainerDied","Data":"30173284b14ae55745dad218452261bc33912a1868e42d98c3729f1881d3e6c7"} Nov 03 02:41:20.244514 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:20.244455 5437 scope.go:122] "RemoveContainer" containerID="1131d4eaeeb8df52191b82138c24643d48e13b6b3dab9cc1546a4675639868ba" Nov 03 02:41:20.253252 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:20.253230 5437 scope.go:122] "RemoveContainer" containerID="1131d4eaeeb8df52191b82138c24643d48e13b6b3dab9cc1546a4675639868ba" Nov 03 02:41:20.253563 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:41:20.253536 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1131d4eaeeb8df52191b82138c24643d48e13b6b3dab9cc1546a4675639868ba\": container with ID starting with 1131d4eaeeb8df52191b82138c24643d48e13b6b3dab9cc1546a4675639868ba not found: ID does not exist" containerID="1131d4eaeeb8df52191b82138c24643d48e13b6b3dab9cc1546a4675639868ba" Nov 03 02:41:20.253663 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:20.253571 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1131d4eaeeb8df52191b82138c24643d48e13b6b3dab9cc1546a4675639868ba"} err="failed to get container status \"1131d4eaeeb8df52191b82138c24643d48e13b6b3dab9cc1546a4675639868ba\": rpc error: code = NotFound desc = could not find container \"1131d4eaeeb8df52191b82138c24643d48e13b6b3dab9cc1546a4675639868ba\": container with ID starting with 1131d4eaeeb8df52191b82138c24643d48e13b6b3dab9cc1546a4675639868ba not found: ID does not exist" Nov 03 02:41:20.416169 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:20.416075 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-0bf34a-66c455cb47-xbxp2"] Nov 03 02:41:20.466499 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:20.466465 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-0bf34a-66c455cb47-xbxp2"] Nov 03 02:41:21.410975 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:21.410931 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:41:22.076626 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:22.076591 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:41:22.088508 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:22.088473 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28d9cb53-e595-4660-9656-0ef00e30d0d4" path="/var/lib/kubelet/pods/28d9cb53-e595-4660-9656-0ef00e30d0d4/volumes" Nov 03 02:41:22.279674 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:22.279623 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:41:22.413339 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:22.413312 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-wgtp9" Nov 03 02:41:22.566674 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:22.566639 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6807c987-7fd9-4b0d-bee4-dab5799ece82-kube-api-access-ks474\" (UniqueName: \"kubernetes.io/projected/6807c987-7fd9-4b0d-bee4-dab5799ece82-kube-api-access-ks474\") pod \"6807c987-7fd9-4b0d-bee4-dab5799ece82\" (UID: \"6807c987-7fd9-4b0d-bee4-dab5799ece82\") " Nov 03 02:41:22.566884 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:22.566693 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6807c987-7fd9-4b0d-bee4-dab5799ece82-istio-token\" (UniqueName: \"kubernetes.io/projected/6807c987-7fd9-4b0d-bee4-dab5799ece82-istio-token\") pod \"6807c987-7fd9-4b0d-bee4-dab5799ece82\" (UID: \"6807c987-7fd9-4b0d-bee4-dab5799ece82\") " Nov 03 02:41:22.566884 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:22.566743 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6807c987-7fd9-4b0d-bee4-dab5799ece82-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6807c987-7fd9-4b0d-bee4-dab5799ece82-istiod-ca-cert\") pod \"6807c987-7fd9-4b0d-bee4-dab5799ece82\" (UID: \"6807c987-7fd9-4b0d-bee4-dab5799ece82\") " Nov 03 02:41:22.566884 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:22.566814 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6807c987-7fd9-4b0d-bee4-dab5799ece82-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6807c987-7fd9-4b0d-bee4-dab5799ece82-istio-envoy\") pod \"6807c987-7fd9-4b0d-bee4-dab5799ece82\" (UID: \"6807c987-7fd9-4b0d-bee4-dab5799ece82\") " Nov 03 02:41:22.567067 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:22.566910 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6807c987-7fd9-4b0d-bee4-dab5799ece82-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6807c987-7fd9-4b0d-bee4-dab5799ece82-credential-socket\") pod \"6807c987-7fd9-4b0d-bee4-dab5799ece82\" (UID: \"6807c987-7fd9-4b0d-bee4-dab5799ece82\") " Nov 03 02:41:22.567067 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:22.566954 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6807c987-7fd9-4b0d-bee4-dab5799ece82-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6807c987-7fd9-4b0d-bee4-dab5799ece82-workload-socket\") pod \"6807c987-7fd9-4b0d-bee4-dab5799ece82\" (UID: \"6807c987-7fd9-4b0d-bee4-dab5799ece82\") " Nov 03 02:41:22.567067 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:22.566997 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6807c987-7fd9-4b0d-bee4-dab5799ece82-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6807c987-7fd9-4b0d-bee4-dab5799ece82-istio-data\") pod \"6807c987-7fd9-4b0d-bee4-dab5799ece82\" (UID: \"6807c987-7fd9-4b0d-bee4-dab5799ece82\") " Nov 03 02:41:22.567067 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:22.567048 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6807c987-7fd9-4b0d-bee4-dab5799ece82-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6807c987-7fd9-4b0d-bee4-dab5799ece82-istio-podinfo\") pod \"6807c987-7fd9-4b0d-bee4-dab5799ece82\" (UID: \"6807c987-7fd9-4b0d-bee4-dab5799ece82\") " Nov 03 02:41:22.567275 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:22.567080 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6807c987-7fd9-4b0d-bee4-dab5799ece82-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6807c987-7fd9-4b0d-bee4-dab5799ece82-workload-certs\") pod \"6807c987-7fd9-4b0d-bee4-dab5799ece82\" (UID: \"6807c987-7fd9-4b0d-bee4-dab5799ece82\") " Nov 03 02:41:22.567275 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:22.567178 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6807c987-7fd9-4b0d-bee4-dab5799ece82-istiod-ca-cert" pod "6807c987-7fd9-4b0d-bee4-dab5799ece82" (UID: "6807c987-7fd9-4b0d-bee4-dab5799ece82"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:41:22.567561 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:22.567307 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6807c987-7fd9-4b0d-bee4-dab5799ece82-workload-socket" pod "6807c987-7fd9-4b0d-bee4-dab5799ece82" (UID: "6807c987-7fd9-4b0d-bee4-dab5799ece82"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:41:22.567561 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:22.567352 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6807c987-7fd9-4b0d-bee4-dab5799ece82-istio-data" pod "6807c987-7fd9-4b0d-bee4-dab5799ece82" (UID: "6807c987-7fd9-4b0d-bee4-dab5799ece82"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:41:22.567561 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:22.567446 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6807c987-7fd9-4b0d-bee4-dab5799ece82-credential-socket" pod "6807c987-7fd9-4b0d-bee4-dab5799ece82" (UID: "6807c987-7fd9-4b0d-bee4-dab5799ece82"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:41:22.567561 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:22.567472 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6807c987-7fd9-4b0d-bee4-dab5799ece82-workload-certs" pod "6807c987-7fd9-4b0d-bee4-dab5799ece82" (UID: "6807c987-7fd9-4b0d-bee4-dab5799ece82"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:41:22.567789 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:22.567684 5437 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6807c987-7fd9-4b0d-bee4-dab5799ece82-workload-certs\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:41:22.567789 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:22.567709 5437 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6807c987-7fd9-4b0d-bee4-dab5799ece82-istiod-ca-cert\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:41:22.567789 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:22.567725 5437 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6807c987-7fd9-4b0d-bee4-dab5799ece82-credential-socket\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:41:22.567789 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:22.567740 5437 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6807c987-7fd9-4b0d-bee4-dab5799ece82-workload-socket\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:41:22.567789 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:22.567754 5437 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6807c987-7fd9-4b0d-bee4-dab5799ece82-istio-data\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:41:22.569038 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:22.569012 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6807c987-7fd9-4b0d-bee4-dab5799ece82-kube-api-access-ks474" pod "6807c987-7fd9-4b0d-bee4-dab5799ece82" (UID: "6807c987-7fd9-4b0d-bee4-dab5799ece82"). InnerVolumeSpecName "kube-api-access-ks474". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:41:22.569038 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:22.569012 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6807c987-7fd9-4b0d-bee4-dab5799ece82-istio-envoy" pod "6807c987-7fd9-4b0d-bee4-dab5799ece82" (UID: "6807c987-7fd9-4b0d-bee4-dab5799ece82"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:41:22.569203 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:22.569048 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6807c987-7fd9-4b0d-bee4-dab5799ece82-istio-token" pod "6807c987-7fd9-4b0d-bee4-dab5799ece82" (UID: "6807c987-7fd9-4b0d-bee4-dab5799ece82"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:41:22.569448 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:22.569424 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6807c987-7fd9-4b0d-bee4-dab5799ece82-istio-podinfo" pod "6807c987-7fd9-4b0d-bee4-dab5799ece82" (UID: "6807c987-7fd9-4b0d-bee4-dab5799ece82"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:41:22.617050 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:22.617016 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:41:22.668674 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:22.668580 5437 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6807c987-7fd9-4b0d-bee4-dab5799ece82-istio-podinfo\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:41:22.668674 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:22.668615 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ks474\" (UniqueName: \"kubernetes.io/projected/6807c987-7fd9-4b0d-bee4-dab5799ece82-kube-api-access-ks474\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:41:22.668674 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:22.668630 5437 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6807c987-7fd9-4b0d-bee4-dab5799ece82-istio-token\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:41:22.668674 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:22.668642 5437 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6807c987-7fd9-4b0d-bee4-dab5799ece82-istio-envoy\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:41:23.003278 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:23.003186 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:41:23.261468 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:23.261377 5437 generic.go:356] "Generic (PLEG): container finished" podID="6807c987-7fd9-4b0d-bee4-dab5799ece82" containerID="421af643089b5b8096e4e1f8fbac7c9e1f8409c5ef2bf3134677aaec3660fc8d" exitCode=0 Nov 03 02:41:23.261468 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:23.261427 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-wgtp9" Nov 03 02:41:23.261468 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:23.261441 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-wgtp9" event={"ID":"6807c987-7fd9-4b0d-bee4-dab5799ece82","Type":"ContainerDied","Data":"421af643089b5b8096e4e1f8fbac7c9e1f8409c5ef2bf3134677aaec3660fc8d"} Nov 03 02:41:23.261716 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:23.261481 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-wgtp9" event={"ID":"6807c987-7fd9-4b0d-bee4-dab5799ece82","Type":"ContainerDied","Data":"f6664f318d0507e42da429d7e25bf7d6ffabf4e0a68812403b4d627642f62ac2"} Nov 03 02:41:23.261716 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:23.261503 5437 scope.go:122] "RemoveContainer" containerID="421af643089b5b8096e4e1f8fbac7c9e1f8409c5ef2bf3134677aaec3660fc8d" Nov 03 02:41:23.271838 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:23.271676 5437 scope.go:122] "RemoveContainer" containerID="421af643089b5b8096e4e1f8fbac7c9e1f8409c5ef2bf3134677aaec3660fc8d" Nov 03 02:41:23.271983 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:41:23.271964 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"421af643089b5b8096e4e1f8fbac7c9e1f8409c5ef2bf3134677aaec3660fc8d\": container with ID starting with 421af643089b5b8096e4e1f8fbac7c9e1f8409c5ef2bf3134677aaec3660fc8d not found: ID does not exist" containerID="421af643089b5b8096e4e1f8fbac7c9e1f8409c5ef2bf3134677aaec3660fc8d" Nov 03 02:41:23.272032 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:23.271992 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"421af643089b5b8096e4e1f8fbac7c9e1f8409c5ef2bf3134677aaec3660fc8d"} err="failed to get container status \"421af643089b5b8096e4e1f8fbac7c9e1f8409c5ef2bf3134677aaec3660fc8d\": rpc error: code = NotFound desc = could not find container \"421af643089b5b8096e4e1f8fbac7c9e1f8409c5ef2bf3134677aaec3660fc8d\": container with ID starting with 421af643089b5b8096e4e1f8fbac7c9e1f8409c5ef2bf3134677aaec3660fc8d not found: ID does not exist" Nov 03 02:41:23.482829 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:23.482782 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-wgtp9"] Nov 03 02:41:23.534439 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:23.534349 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-wgtp9"] Nov 03 02:41:23.636307 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:23.636270 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:41:24.087359 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:24.087314 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6807c987-7fd9-4b0d-bee4-dab5799ece82" path="/var/lib/kubelet/pods/6807c987-7fd9-4b0d-bee4-dab5799ece82/volumes" Nov 03 02:41:24.125254 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:24.125220 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:41:27.517222 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:27.517168 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:41:27.746351 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:27.746308 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:41:29.713913 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:29.713868 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:41:32.381347 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:32.381312 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:41:33.631826 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:33.631782 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:41:33.899671 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:33.899582 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-9da431-756c59d995-5p6n9"] Nov 03 02:41:34.007628 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:34.007586 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-9da431-69447f989f-7qws7"] Nov 03 02:41:34.291430 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:34.291322 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:41:34.503323 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:34.503281 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:41:35.900376 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:35.900324 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:41:36.115093 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:36.115058 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:41:37.555923 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:37.555880 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:41:38.900096 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:38.900033 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-9da431-756c59d995-5p6n9" podUID="a7e0c5cd-7758-4a54-9a9f-18797ffe2568" containerName="payload-processing" containerID="cri-o://630e37c39b6a34b7dd731aa34f7b824396861196cfef1f3328bac832b4f1121a" gracePeriod=25 Nov 03 02:41:39.008467 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:39.008356 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-9da431-69447f989f-7qws7" podUID="952ba771-48ba-43ed-8ceb-290a0e79ebdb" containerName="payload-pre-processing" containerID="cri-o://22f5233c31242f73f0b78b19566ccc8d3cc80c7d56f265d0dd229484f2d1c3bc" gracePeriod=25 Nov 03 02:41:39.804169 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:39.804136 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:41:42.380631 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:42.380589 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:41:43.693961 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:43.693921 5437 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-9da431-756c59d995-5p6n9" podUID="a7e0c5cd-7758-4a54-9a9f-18797ffe2568" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 02:41:43.694414 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:43.693925 5437 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-9da431-69447f989f-7qws7" podUID="952ba771-48ba-43ed-8ceb-290a0e79ebdb" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 02:41:44.034265 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:41:44.034182 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 02:41:46.198987 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:46.198947 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:41:46.421248 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:46.421210 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-9da431-79b57b969c-4n9ts"] Nov 03 02:41:46.421492 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:46.421452 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9da431-79b57b969c-4n9ts" podUID="f277337a-c920-4c20-a029-215d4ee996f9" containerName="maas-api" containerID="cri-o://b8132051b75cad7e7ea9ecca9391a203e6876b3c85b236a6930b556b9b35cf62" gracePeriod=30 Nov 03 02:41:46.579265 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:46.579229 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:41:46.683489 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:46.683446 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9da431-79b57b969c-4n9ts" Nov 03 02:41:46.718485 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:46.718446 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f277337a-c920-4c20-a029-215d4ee996f9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f277337a-c920-4c20-a029-215d4ee996f9-maas-api-tls\") pod \"f277337a-c920-4c20-a029-215d4ee996f9\" (UID: \"f277337a-c920-4c20-a029-215d4ee996f9\") " Nov 03 02:41:46.718702 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:46.718514 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f277337a-c920-4c20-a029-215d4ee996f9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f277337a-c920-4c20-a029-215d4ee996f9-metrics-certs\") pod \"f277337a-c920-4c20-a029-215d4ee996f9\" (UID: \"f277337a-c920-4c20-a029-215d4ee996f9\") " Nov 03 02:41:46.718702 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:46.718544 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f277337a-c920-4c20-a029-215d4ee996f9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f277337a-c920-4c20-a029-215d4ee996f9-openshift-service-ca\") pod \"f277337a-c920-4c20-a029-215d4ee996f9\" (UID: \"f277337a-c920-4c20-a029-215d4ee996f9\") " Nov 03 02:41:46.718702 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:46.718614 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f277337a-c920-4c20-a029-215d4ee996f9-kube-api-access-g682j\" (UniqueName: \"kubernetes.io/projected/f277337a-c920-4c20-a029-215d4ee996f9-kube-api-access-g682j\") pod \"f277337a-c920-4c20-a029-215d4ee996f9\" (UID: \"f277337a-c920-4c20-a029-215d4ee996f9\") " Nov 03 02:41:46.719000 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:46.718967 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f277337a-c920-4c20-a029-215d4ee996f9-openshift-service-ca" pod "f277337a-c920-4c20-a029-215d4ee996f9" (UID: "f277337a-c920-4c20-a029-215d4ee996f9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:41:46.720846 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:46.720821 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f277337a-c920-4c20-a029-215d4ee996f9-maas-api-tls" pod "f277337a-c920-4c20-a029-215d4ee996f9" (UID: "f277337a-c920-4c20-a029-215d4ee996f9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:41:46.720976 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:46.720860 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f277337a-c920-4c20-a029-215d4ee996f9-metrics-certs" pod "f277337a-c920-4c20-a029-215d4ee996f9" (UID: "f277337a-c920-4c20-a029-215d4ee996f9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:41:46.720976 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:46.720929 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f277337a-c920-4c20-a029-215d4ee996f9-kube-api-access-g682j" pod "f277337a-c920-4c20-a029-215d4ee996f9" (UID: "f277337a-c920-4c20-a029-215d4ee996f9"). InnerVolumeSpecName "kube-api-access-g682j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:41:46.819524 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:46.819490 5437 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f277337a-c920-4c20-a029-215d4ee996f9-maas-api-tls\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:41:46.819524 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:46.819522 5437 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f277337a-c920-4c20-a029-215d4ee996f9-metrics-certs\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:41:46.819748 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:46.819535 5437 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f277337a-c920-4c20-a029-215d4ee996f9-openshift-service-ca\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:41:46.819748 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:46.819548 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g682j\" (UniqueName: \"kubernetes.io/projected/f277337a-c920-4c20-a029-215d4ee996f9-kube-api-access-g682j\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:41:47.387257 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:47.387216 5437 generic.go:356] "Generic (PLEG): container finished" podID="f277337a-c920-4c20-a029-215d4ee996f9" containerID="b8132051b75cad7e7ea9ecca9391a203e6876b3c85b236a6930b556b9b35cf62" exitCode=0 Nov 03 02:41:47.387786 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:47.387270 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9da431-79b57b969c-4n9ts" Nov 03 02:41:47.387786 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:47.387302 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9da431-79b57b969c-4n9ts" event={"ID":"f277337a-c920-4c20-a029-215d4ee996f9","Type":"ContainerDied","Data":"b8132051b75cad7e7ea9ecca9391a203e6876b3c85b236a6930b556b9b35cf62"} Nov 03 02:41:47.387786 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:47.387386 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9da431-79b57b969c-4n9ts" event={"ID":"f277337a-c920-4c20-a029-215d4ee996f9","Type":"ContainerDied","Data":"8633c0fa33400de290396a94fe48b8fabfd43e5b2af3a22d8d1c35c2a5423720"} Nov 03 02:41:47.387786 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:47.387404 5437 scope.go:122] "RemoveContainer" containerID="b8132051b75cad7e7ea9ecca9391a203e6876b3c85b236a6930b556b9b35cf62" Nov 03 02:41:47.398401 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:47.398374 5437 scope.go:122] "RemoveContainer" containerID="b8132051b75cad7e7ea9ecca9391a203e6876b3c85b236a6930b556b9b35cf62" Nov 03 02:41:47.398753 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:41:47.398731 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8132051b75cad7e7ea9ecca9391a203e6876b3c85b236a6930b556b9b35cf62\": container with ID starting with b8132051b75cad7e7ea9ecca9391a203e6876b3c85b236a6930b556b9b35cf62 not found: ID does not exist" containerID="b8132051b75cad7e7ea9ecca9391a203e6876b3c85b236a6930b556b9b35cf62" Nov 03 02:41:47.398826 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:47.398762 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8132051b75cad7e7ea9ecca9391a203e6876b3c85b236a6930b556b9b35cf62"} err="failed to get container status \"b8132051b75cad7e7ea9ecca9391a203e6876b3c85b236a6930b556b9b35cf62\": rpc error: code = NotFound desc = could not find container \"b8132051b75cad7e7ea9ecca9391a203e6876b3c85b236a6930b556b9b35cf62\": container with ID starting with b8132051b75cad7e7ea9ecca9391a203e6876b3c85b236a6930b556b9b35cf62 not found: ID does not exist" Nov 03 02:41:47.559915 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:47.559869 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-9da431-79b57b969c-4n9ts"] Nov 03 02:41:47.610483 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:47.610448 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-9da431-79b57b969c-4n9ts"] Nov 03 02:41:48.088046 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:48.088011 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f277337a-c920-4c20-a029-215d4ee996f9" path="/var/lib/kubelet/pods/f277337a-c920-4c20-a029-215d4ee996f9/volumes" Nov 03 02:41:48.285604 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:48.285569 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:41:51.762243 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:51.762204 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:41:51.960927 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:51.960858 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:41:53.079147 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:53.079106 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:41:53.694352 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:53.694315 5437 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-9da431-756c59d995-5p6n9" podUID="a7e0c5cd-7758-4a54-9a9f-18797ffe2568" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 02:41:53.694562 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:53.694352 5437 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-9da431-69447f989f-7qws7" podUID="952ba771-48ba-43ed-8ceb-290a0e79ebdb" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 02:41:57.054959 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:57.054919 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:41:57.310557 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:57.310469 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:41:57.878295 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:57.878258 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:41:58.946593 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:58.946558 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:41:59.145194 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:59.145156 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:41:59.157628 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:59.157557 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-9da431-756c59d995-5p6n9" Nov 03 02:41:59.238865 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:59.238833 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7e0c5cd-7758-4a54-9a9f-18797ffe2568-kube-api-access-x4df6\" (UniqueName: \"kubernetes.io/projected/a7e0c5cd-7758-4a54-9a9f-18797ffe2568-kube-api-access-x4df6\") pod \"a7e0c5cd-7758-4a54-9a9f-18797ffe2568\" (UID: \"a7e0c5cd-7758-4a54-9a9f-18797ffe2568\") " Nov 03 02:41:59.239058 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:59.238951 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7e0c5cd-7758-4a54-9a9f-18797ffe2568-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7e0c5cd-7758-4a54-9a9f-18797ffe2568-plugins-config-volume\") pod \"a7e0c5cd-7758-4a54-9a9f-18797ffe2568\" (UID: \"a7e0c5cd-7758-4a54-9a9f-18797ffe2568\") " Nov 03 02:41:59.240398 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:59.239323 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7e0c5cd-7758-4a54-9a9f-18797ffe2568-plugins-config-volume" pod "a7e0c5cd-7758-4a54-9a9f-18797ffe2568" (UID: "a7e0c5cd-7758-4a54-9a9f-18797ffe2568"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:41:59.240398 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:59.239477 5437 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7e0c5cd-7758-4a54-9a9f-18797ffe2568-plugins-config-volume\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:41:59.243595 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:59.243560 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7e0c5cd-7758-4a54-9a9f-18797ffe2568-kube-api-access-x4df6" pod "a7e0c5cd-7758-4a54-9a9f-18797ffe2568" (UID: "a7e0c5cd-7758-4a54-9a9f-18797ffe2568"). InnerVolumeSpecName "kube-api-access-x4df6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:41:59.258407 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:59.258381 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-9da431-69447f989f-7qws7" Nov 03 02:41:59.340008 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:59.339968 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/952ba771-48ba-43ed-8ceb-290a0e79ebdb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/952ba771-48ba-43ed-8ceb-290a0e79ebdb-plugins-config-volume\") pod \"952ba771-48ba-43ed-8ceb-290a0e79ebdb\" (UID: \"952ba771-48ba-43ed-8ceb-290a0e79ebdb\") " Nov 03 02:41:59.340200 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:59.340036 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/952ba771-48ba-43ed-8ceb-290a0e79ebdb-kube-api-access-zxvbw\" (UniqueName: \"kubernetes.io/projected/952ba771-48ba-43ed-8ceb-290a0e79ebdb-kube-api-access-zxvbw\") pod \"952ba771-48ba-43ed-8ceb-290a0e79ebdb\" (UID: \"952ba771-48ba-43ed-8ceb-290a0e79ebdb\") " Nov 03 02:41:59.340381 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:59.340350 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4df6\" (UniqueName: \"kubernetes.io/projected/a7e0c5cd-7758-4a54-9a9f-18797ffe2568-kube-api-access-x4df6\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:41:59.340517 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:59.340481 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/952ba771-48ba-43ed-8ceb-290a0e79ebdb-plugins-config-volume" pod "952ba771-48ba-43ed-8ceb-290a0e79ebdb" (UID: "952ba771-48ba-43ed-8ceb-290a0e79ebdb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:41:59.342458 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:59.342429 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/952ba771-48ba-43ed-8ceb-290a0e79ebdb-kube-api-access-zxvbw" pod "952ba771-48ba-43ed-8ceb-290a0e79ebdb" (UID: "952ba771-48ba-43ed-8ceb-290a0e79ebdb"). InnerVolumeSpecName "kube-api-access-zxvbw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:41:59.401134 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:59.401095 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:41:59.441233 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:59.441144 5437 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/952ba771-48ba-43ed-8ceb-290a0e79ebdb-plugins-config-volume\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:41:59.441233 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:59.441173 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxvbw\" (UniqueName: \"kubernetes.io/projected/952ba771-48ba-43ed-8ceb-290a0e79ebdb-kube-api-access-zxvbw\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:41:59.449841 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:59.449809 5437 generic.go:356] "Generic (PLEG): container finished" podID="952ba771-48ba-43ed-8ceb-290a0e79ebdb" containerID="22f5233c31242f73f0b78b19566ccc8d3cc80c7d56f265d0dd229484f2d1c3bc" exitCode=0 Nov 03 02:41:59.450043 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:59.449856 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-9da431-69447f989f-7qws7" Nov 03 02:41:59.450043 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:59.449891 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-9da431-69447f989f-7qws7" event={"ID":"952ba771-48ba-43ed-8ceb-290a0e79ebdb","Type":"ContainerDied","Data":"22f5233c31242f73f0b78b19566ccc8d3cc80c7d56f265d0dd229484f2d1c3bc"} Nov 03 02:41:59.450043 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:59.449947 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-9da431-69447f989f-7qws7" event={"ID":"952ba771-48ba-43ed-8ceb-290a0e79ebdb","Type":"ContainerDied","Data":"137edb5df23072a637f814d4bac4ec674254fd828bafd08687bd7d678112d006"} Nov 03 02:41:59.450043 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:59.449977 5437 scope.go:122] "RemoveContainer" containerID="22f5233c31242f73f0b78b19566ccc8d3cc80c7d56f265d0dd229484f2d1c3bc" Nov 03 02:41:59.451431 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:59.451410 5437 generic.go:356] "Generic (PLEG): container finished" podID="a7e0c5cd-7758-4a54-9a9f-18797ffe2568" containerID="630e37c39b6a34b7dd731aa34f7b824396861196cfef1f3328bac832b4f1121a" exitCode=0 Nov 03 02:41:59.451523 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:59.451453 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-9da431-756c59d995-5p6n9" Nov 03 02:41:59.451596 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:59.451455 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-9da431-756c59d995-5p6n9" event={"ID":"a7e0c5cd-7758-4a54-9a9f-18797ffe2568","Type":"ContainerDied","Data":"630e37c39b6a34b7dd731aa34f7b824396861196cfef1f3328bac832b4f1121a"} Nov 03 02:41:59.451596 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:59.451590 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-9da431-756c59d995-5p6n9" event={"ID":"a7e0c5cd-7758-4a54-9a9f-18797ffe2568","Type":"ContainerDied","Data":"ef810443ae0061bd04f3f645c99bc1f634d8404c003612cbb52dd1af700dc233"} Nov 03 02:41:59.459641 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:59.459537 5437 scope.go:122] "RemoveContainer" containerID="22f5233c31242f73f0b78b19566ccc8d3cc80c7d56f265d0dd229484f2d1c3bc" Nov 03 02:41:59.460038 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:41:59.460006 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22f5233c31242f73f0b78b19566ccc8d3cc80c7d56f265d0dd229484f2d1c3bc\": container with ID starting with 22f5233c31242f73f0b78b19566ccc8d3cc80c7d56f265d0dd229484f2d1c3bc not found: ID does not exist" containerID="22f5233c31242f73f0b78b19566ccc8d3cc80c7d56f265d0dd229484f2d1c3bc" Nov 03 02:41:59.460142 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:59.460045 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22f5233c31242f73f0b78b19566ccc8d3cc80c7d56f265d0dd229484f2d1c3bc"} err="failed to get container status \"22f5233c31242f73f0b78b19566ccc8d3cc80c7d56f265d0dd229484f2d1c3bc\": rpc error: code = NotFound desc = could not find container \"22f5233c31242f73f0b78b19566ccc8d3cc80c7d56f265d0dd229484f2d1c3bc\": container with ID starting with 22f5233c31242f73f0b78b19566ccc8d3cc80c7d56f265d0dd229484f2d1c3bc not found: ID does not exist" Nov 03 02:41:59.460142 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:59.460070 5437 scope.go:122] "RemoveContainer" containerID="630e37c39b6a34b7dd731aa34f7b824396861196cfef1f3328bac832b4f1121a" Nov 03 02:41:59.469040 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:59.469021 5437 scope.go:122] "RemoveContainer" containerID="630e37c39b6a34b7dd731aa34f7b824396861196cfef1f3328bac832b4f1121a" Nov 03 02:41:59.469346 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:41:59.469326 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"630e37c39b6a34b7dd731aa34f7b824396861196cfef1f3328bac832b4f1121a\": container with ID starting with 630e37c39b6a34b7dd731aa34f7b824396861196cfef1f3328bac832b4f1121a not found: ID does not exist" containerID="630e37c39b6a34b7dd731aa34f7b824396861196cfef1f3328bac832b4f1121a" Nov 03 02:41:59.469446 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:59.469355 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"630e37c39b6a34b7dd731aa34f7b824396861196cfef1f3328bac832b4f1121a"} err="failed to get container status \"630e37c39b6a34b7dd731aa34f7b824396861196cfef1f3328bac832b4f1121a\": rpc error: code = NotFound desc = could not find container \"630e37c39b6a34b7dd731aa34f7b824396861196cfef1f3328bac832b4f1121a\": container with ID starting with 630e37c39b6a34b7dd731aa34f7b824396861196cfef1f3328bac832b4f1121a not found: ID does not exist" Nov 03 02:41:59.619639 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:59.619604 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-9da431-69447f989f-7qws7"] Nov 03 02:41:59.623226 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:59.623196 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-9da431-69447f989f-7qws7"] Nov 03 02:41:59.737230 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:59.737143 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-9da431-756c59d995-5p6n9"] Nov 03 02:41:59.740885 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:41:59.740855 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-9da431-756c59d995-5p6n9"] Nov 03 02:42:00.088068 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:42:00.088030 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="952ba771-48ba-43ed-8ceb-290a0e79ebdb" path="/var/lib/kubelet/pods/952ba771-48ba-43ed-8ceb-290a0e79ebdb/volumes" Nov 03 02:42:00.088478 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:42:00.088450 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7e0c5cd-7758-4a54-9a9f-18797ffe2568" path="/var/lib/kubelet/pods/a7e0c5cd-7758-4a54-9a9f-18797ffe2568/volumes" Nov 03 02:42:02.036119 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:42:02.036074 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:42:02.537774 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:42:02.537737 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:42:02.937600 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:42:02.937564 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:42:04.235380 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:42:04.235328 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:42:04.434927 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:42:04.434894 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:42:05.945882 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:42:05.945840 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:42:06.153331 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:42:06.153283 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:42:07.896974 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:42:07.896931 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:42:09.279352 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:42:09.279308 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:42:12.678349 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:42:12.678311 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:42:14.034723 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:42:14.034675 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 02:42:17.477468 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:42:17.477429 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:42:22.284487 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:42:22.284452 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:42:26.465530 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:42:26.465492 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:42:26.718006 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:42:26.717916 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:42:27.090651 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:42:27.090617 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:42:32.169992 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:42:32.169954 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:42:35.426973 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:42:35.426930 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:42:35.635253 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:42:35.635220 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:42:36.682204 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:42:36.682164 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:42:41.488775 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:42:41.488741 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:42:44.034787 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:42:44.034749 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 02:42:46.289789 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:42:46.289756 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:42:51.101894 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:42:51.101805 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:42:55.886246 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:42:55.886209 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:43:00.688837 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:43:00.688801 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:43:04.631793 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:43:04.631756 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:43:05.501829 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:43:05.501787 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:43:09.622123 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:43:09.622082 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:43:10.273873 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:43:10.273834 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:43:14.034481 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:43:14.034442 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 02:43:15.079440 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:43:15.079400 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:43:19.877939 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:43:19.877891 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:43:24.675657 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:43:24.675613 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:43:29.486800 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:43:29.486764 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:43:34.283225 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:43:34.283190 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:43:39.033756 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:43:39.033708 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:43:43.888172 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:43:43.888137 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:43:44.033937 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:43:44.033887 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 02:43:45.123217 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:43:45.123179 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:43:48.684868 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:43:48.684821 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:43:53.564447 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:43:53.564404 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:43:58.381694 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:43:58.381654 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:44:03.386005 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:44:03.385968 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:44:04.991262 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:44:04.991222 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:44:08.265107 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:44:08.265073 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:44:09.952051 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:44:09.952008 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:44:09.961786 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:44:09.961754 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7002-d660ba"] Nov 03 02:44:09.962180 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:44:09.962167 5437 cpu_manager.go:414] "containerMap: removing container" podUID="952ba771-48ba-43ed-8ceb-290a0e79ebdb" containerName="payload-pre-processing" Nov 03 02:44:09.962225 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:44:09.962182 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="952ba771-48ba-43ed-8ceb-290a0e79ebdb" containerName="payload-pre-processing" Nov 03 02:44:09.962225 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:44:09.962195 5437 cpu_manager.go:414] "containerMap: removing container" podUID="b5d7ce00-728b-437c-9fbd-524d8c4697fc" containerName="payload-pre-processing" Nov 03 02:44:09.962225 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:44:09.962201 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5d7ce00-728b-437c-9fbd-524d8c4697fc" containerName="payload-pre-processing" Nov 03 02:44:09.962225 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:44:09.962211 5437 cpu_manager.go:414] "containerMap: removing container" podUID="4cfca4a1-0da6-4637-bc48-8e65776b9ca9" containerName="main" Nov 03 02:44:09.962225 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:44:09.962216 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cfca4a1-0da6-4637-bc48-8e65776b9ca9" containerName="main" Nov 03 02:44:09.962225 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:44:09.962225 5437 cpu_manager.go:414] "containerMap: removing container" podUID="bcc4bb00-2f9e-4f38-a331-f725729873f0" containerName="maas-api" Nov 03 02:44:09.962225 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:44:09.962229 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bcc4bb00-2f9e-4f38-a331-f725729873f0" containerName="maas-api" Nov 03 02:44:09.962485 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:44:09.962238 5437 cpu_manager.go:414] "containerMap: removing container" podUID="c46dca02-6036-46de-b0f3-df796d231f62" containerName="main" Nov 03 02:44:09.962485 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:44:09.962243 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c46dca02-6036-46de-b0f3-df796d231f62" containerName="main" Nov 03 02:44:09.962485 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:44:09.962250 5437 cpu_manager.go:414] "containerMap: removing container" podUID="6807c987-7fd9-4b0d-bee4-dab5799ece82" containerName="istio-proxy" Nov 03 02:44:09.962485 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:44:09.962255 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6807c987-7fd9-4b0d-bee4-dab5799ece82" containerName="istio-proxy" Nov 03 02:44:09.962485 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:44:09.962261 5437 cpu_manager.go:414] "containerMap: removing container" podUID="8374d59d-7bed-4b5d-b887-013569604481" containerName="payload-processing" Nov 03 02:44:09.962485 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:44:09.962266 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8374d59d-7bed-4b5d-b887-013569604481" containerName="payload-processing" Nov 03 02:44:09.962485 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:44:09.962274 5437 cpu_manager.go:414] "containerMap: removing container" podUID="85af418f-1d3c-44dd-803b-fd73c7e18b0d" containerName="payload-processing" Nov 03 02:44:09.962485 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:44:09.962279 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85af418f-1d3c-44dd-803b-fd73c7e18b0d" containerName="payload-processing" Nov 03 02:44:09.962485 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:44:09.962287 5437 cpu_manager.go:414] "containerMap: removing container" podUID="8a87f809-0b26-480e-bb37-ff234a40c5c5" containerName="main" Nov 03 02:44:09.962485 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:44:09.962291 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a87f809-0b26-480e-bb37-ff234a40c5c5" containerName="main" Nov 03 02:44:09.962485 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:44:09.962298 5437 cpu_manager.go:414] "containerMap: removing container" podUID="f277337a-c920-4c20-a029-215d4ee996f9" containerName="maas-api" Nov 03 02:44:09.962485 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:44:09.962302 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f277337a-c920-4c20-a029-215d4ee996f9" containerName="maas-api" Nov 03 02:44:09.962485 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:44:09.962315 5437 cpu_manager.go:414] "containerMap: removing container" podUID="21b04f72-d240-4ebb-860b-c80607e0d94d" containerName="maas-api" Nov 03 02:44:09.962485 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:44:09.962321 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21b04f72-d240-4ebb-860b-c80607e0d94d" containerName="maas-api" Nov 03 02:44:09.962485 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:44:09.962329 5437 cpu_manager.go:414] "containerMap: removing container" podUID="6050630e-79eb-49c9-bb97-4cd9e9309ffa" containerName="payload-processing" Nov 03 02:44:09.962485 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:44:09.962333 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6050630e-79eb-49c9-bb97-4cd9e9309ffa" containerName="payload-processing" Nov 03 02:44:09.962485 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:44:09.962342 5437 cpu_manager.go:414] "containerMap: removing container" podUID="1f9f6115-560b-451b-ba0d-e66e67ca55eb" containerName="maas-api" Nov 03 02:44:09.962485 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:44:09.962347 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f9f6115-560b-451b-ba0d-e66e67ca55eb" containerName="maas-api" Nov 03 02:44:09.962485 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:44:09.962356 5437 cpu_manager.go:414] "containerMap: removing container" podUID="80edf8e3-195e-4ccf-b826-27adbf5504ef" containerName="istio-proxy" Nov 03 02:44:09.962485 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:44:09.962377 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80edf8e3-195e-4ccf-b826-27adbf5504ef" containerName="istio-proxy" Nov 03 02:44:09.962485 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:44:09.962386 5437 cpu_manager.go:414] "containerMap: removing container" podUID="80edf8e3-195e-4ccf-b826-27adbf5504ef" containerName="istio-proxy" Nov 03 02:44:09.962485 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:44:09.962391 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80edf8e3-195e-4ccf-b826-27adbf5504ef" containerName="istio-proxy" Nov 03 02:44:09.962485 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:44:09.962398 5437 cpu_manager.go:414] "containerMap: removing container" podUID="615071b1-f5f9-4d03-90b8-1210c450902d" containerName="istio-proxy" Nov 03 02:44:09.962485 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:44:09.962402 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="615071b1-f5f9-4d03-90b8-1210c450902d" containerName="istio-proxy" Nov 03 02:44:09.962485 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:44:09.962409 5437 cpu_manager.go:414] "containerMap: removing container" podUID="a7e0c5cd-7758-4a54-9a9f-18797ffe2568" containerName="payload-processing" Nov 03 02:44:09.962485 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:44:09.962413 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7e0c5cd-7758-4a54-9a9f-18797ffe2568" containerName="payload-processing" Nov 03 02:44:09.962485 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:44:09.962423 5437 cpu_manager.go:414] "containerMap: removing container" podUID="28d9cb53-e595-4660-9656-0ef00e30d0d4" containerName="payload-pre-processing" Nov 03 02:44:09.962485 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:44:09.962427 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28d9cb53-e595-4660-9656-0ef00e30d0d4" containerName="payload-pre-processing" Nov 03 02:44:09.962485 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:44:09.962490 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="6050630e-79eb-49c9-bb97-4cd9e9309ffa" containerName="payload-processing" Nov 03 02:44:09.963469 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:44:09.962501 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f9f6115-560b-451b-ba0d-e66e67ca55eb" containerName="maas-api" Nov 03 02:44:09.963469 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:44:09.962507 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="80edf8e3-195e-4ccf-b826-27adbf5504ef" containerName="istio-proxy" Nov 03 02:44:09.963469 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:44:09.962513 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="80edf8e3-195e-4ccf-b826-27adbf5504ef" containerName="istio-proxy" Nov 03 02:44:09.963469 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:44:09.962519 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="615071b1-f5f9-4d03-90b8-1210c450902d" containerName="istio-proxy" Nov 03 02:44:09.963469 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:44:09.962525 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7e0c5cd-7758-4a54-9a9f-18797ffe2568" containerName="payload-processing" Nov 03 02:44:09.963469 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:44:09.962531 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="28d9cb53-e595-4660-9656-0ef00e30d0d4" containerName="payload-pre-processing" Nov 03 02:44:09.963469 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:44:09.962538 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="952ba771-48ba-43ed-8ceb-290a0e79ebdb" containerName="payload-pre-processing" Nov 03 02:44:09.963469 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:44:09.962542 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5d7ce00-728b-437c-9fbd-524d8c4697fc" containerName="payload-pre-processing" Nov 03 02:44:09.963469 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:44:09.962548 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cfca4a1-0da6-4637-bc48-8e65776b9ca9" containerName="main" Nov 03 02:44:09.963469 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:44:09.962553 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="bcc4bb00-2f9e-4f38-a331-f725729873f0" containerName="maas-api" Nov 03 02:44:09.963469 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:44:09.962558 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="c46dca02-6036-46de-b0f3-df796d231f62" containerName="main" Nov 03 02:44:09.963469 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:44:09.962563 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="6807c987-7fd9-4b0d-bee4-dab5799ece82" containerName="istio-proxy" Nov 03 02:44:09.963469 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:44:09.962568 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="8374d59d-7bed-4b5d-b887-013569604481" containerName="payload-processing" Nov 03 02:44:09.963469 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:44:09.962574 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="85af418f-1d3c-44dd-803b-fd73c7e18b0d" containerName="payload-processing" Nov 03 02:44:09.963469 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:44:09.962579 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a87f809-0b26-480e-bb37-ff234a40c5c5" containerName="main" Nov 03 02:44:09.963469 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:44:09.962585 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="f277337a-c920-4c20-a029-215d4ee996f9" containerName="maas-api" Nov 03 02:44:09.963469 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:44:09.962590 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="21b04f72-d240-4ebb-860b-c80607e0d94d" containerName="maas-api" Nov 03 02:44:09.965795 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:44:09.965778 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7002-d660ba" Nov 03 02:44:10.002311 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:44:10.002270 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzsgg\" (UniqueName: \"kubernetes.io/projected/6a5c4acd-245e-4b64-a146-dcc897eb8b8c-kube-api-access-pzsgg\") pod \"test-curl-7002-d660ba\" (UID: \"6a5c4acd-245e-4b64-a146-dcc897eb8b8c\") " pod="opendatahub/test-curl-7002-d660ba" Nov 03 02:44:10.017592 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:44:10.017560 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7002-d660ba"] Nov 03 02:44:10.067607 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:44:10.067581 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-882bv\"" Nov 03 02:44:10.103418 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:44:10.103381 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzsgg\" (UniqueName: \"kubernetes.io/projected/6a5c4acd-245e-4b64-a146-dcc897eb8b8c-kube-api-access-pzsgg\") pod \"test-curl-7002-d660ba\" (UID: \"6a5c4acd-245e-4b64-a146-dcc897eb8b8c\") " pod="opendatahub/test-curl-7002-d660ba" Nov 03 02:44:10.166730 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:44:10.166694 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzsgg\" (UniqueName: \"kubernetes.io/projected/6a5c4acd-245e-4b64-a146-dcc897eb8b8c-kube-api-access-pzsgg\") pod \"test-curl-7002-d660ba\" (UID: \"6a5c4acd-245e-4b64-a146-dcc897eb8b8c\") " pod="opendatahub/test-curl-7002-d660ba" Nov 03 02:44:10.278142 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:44:10.278059 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7002-d660ba" Nov 03 02:44:10.647486 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:44:10.647451 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7002-d660ba"] Nov 03 02:44:11.103262 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:44:11.103219 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7002-d660ba" event={"ID":"6a5c4acd-245e-4b64-a146-dcc897eb8b8c","Type":"ContainerStarted","Data":"879bce6bf367af385675c69ef7bf8e26f124f5c698807dc3f2b80d883e623d1f"} Nov 03 02:44:11.103262 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:44:11.103264 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7002-d660ba" event={"ID":"6a5c4acd-245e-4b64-a146-dcc897eb8b8c","Type":"ContainerStarted","Data":"a9a5d32aba1c3f04e497edd5b0b131e0f6581fed09dfa29ca76307f8624f3591"} Nov 03 02:44:11.221862 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:44:11.221800 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7002-d660ba" podStartSLOduration=2.221787016 podStartE2EDuration="2.221787016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:44:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:44:11.22010751 +0000 UTC m=+5133.249589805" watchObservedRunningTime="2026-11-03 02:44:11.221787016 +0000 UTC m=+5133.251269307" Nov 03 02:44:12.765534 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:44:12.765499 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7002-d660ba"] Nov 03 02:44:12.767209 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:44:12.767181 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7002-d660ba"] Nov 03 02:44:12.912238 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:44:12.912201 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7002-2c3c40"] Nov 03 02:44:12.912647 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:44:12.912632 5437 cpu_manager.go:414] "containerMap: removing container" podUID="6a5c4acd-245e-4b64-a146-dcc897eb8b8c" containerName="test-curl-7002-d660ba" Nov 03 02:44:12.912703 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:44:12.912648 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a5c4acd-245e-4b64-a146-dcc897eb8b8c" containerName="test-curl-7002-d660ba" Nov 03 02:44:12.912766 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:44:12.912756 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a5c4acd-245e-4b64-a146-dcc897eb8b8c" containerName="test-curl-7002-d660ba" Nov 03 02:44:12.915900 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:44:12.915879 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7002-2c3c40" Nov 03 02:44:12.968627 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:44:12.968582 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7002-2c3c40"] Nov 03 02:44:13.036863 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:44:13.036775 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bss4q\" (UniqueName: \"kubernetes.io/projected/04fc53e7-621c-4cf6-a890-2df029725785-kube-api-access-bss4q\") pod \"test-curl-7002-2c3c40\" (UID: \"04fc53e7-621c-4cf6-a890-2df029725785\") " pod="opendatahub/test-curl-7002-2c3c40" Nov 03 02:44:13.065217 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:44:13.065173 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:44:13.113896 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:44:13.113857 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7002-d660ba" podUID="6a5c4acd-245e-4b64-a146-dcc897eb8b8c" containerName="test-curl-7002-d660ba" containerID="cri-o://879bce6bf367af385675c69ef7bf8e26f124f5c698807dc3f2b80d883e623d1f" gracePeriod=2 Nov 03 02:44:13.138213 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:44:13.138174 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bss4q\" (UniqueName: \"kubernetes.io/projected/04fc53e7-621c-4cf6-a890-2df029725785-kube-api-access-bss4q\") pod \"test-curl-7002-2c3c40\" (UID: \"04fc53e7-621c-4cf6-a890-2df029725785\") " pod="opendatahub/test-curl-7002-2c3c40" Nov 03 02:44:13.167086 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:44:13.167046 5437 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7002-d660ba\" is forbidden: User \"system:node:ip-10-0-136-16.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-16.us-west-2.compute.internal' and this object" podUID="6a5c4acd-245e-4b64-a146-dcc897eb8b8c" pod="opendatahub/test-curl-7002-d660ba" Nov 03 02:44:13.201299 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:44:13.201263 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bss4q\" (UniqueName: \"kubernetes.io/projected/04fc53e7-621c-4cf6-a890-2df029725785-kube-api-access-bss4q\") pod \"test-curl-7002-2c3c40\" (UID: \"04fc53e7-621c-4cf6-a890-2df029725785\") " pod="opendatahub/test-curl-7002-2c3c40" Nov 03 02:44:13.228187 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:44:13.228154 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7002-2c3c40" Nov 03 02:44:13.592298 ip-10-0-136-16 kubenswrapper[5437]: W1103 02:44:13.592267 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod04fc53e7_621c_4cf6_a890_2df029725785.slice/crio-12355cf6a2ad92484d164a5accc927449188abb1a0b296ae8a2d9d7906ea7ce8 WatchSource:0}: Error finding container 12355cf6a2ad92484d164a5accc927449188abb1a0b296ae8a2d9d7906ea7ce8: Status 404 returned error can't find the container with id 12355cf6a2ad92484d164a5accc927449188abb1a0b296ae8a2d9d7906ea7ce8 Nov 03 02:44:13.594621 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:44:13.594593 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7002-2c3c40"] Nov 03 02:44:14.034189 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:44:14.034100 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 02:44:14.119545 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:44:14.119502 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7002-2c3c40" event={"ID":"04fc53e7-621c-4cf6-a890-2df029725785","Type":"ContainerStarted","Data":"7121a63b872d89388c2fee718faf103d5d8e75ea9510a2abc1c2332176431ae3"} Nov 03 02:44:14.119746 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:44:14.119557 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7002-2c3c40" event={"ID":"04fc53e7-621c-4cf6-a890-2df029725785","Type":"ContainerStarted","Data":"12355cf6a2ad92484d164a5accc927449188abb1a0b296ae8a2d9d7906ea7ce8"} Nov 03 02:44:14.238624 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:44:14.238570 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7002-2c3c40" podStartSLOduration=2.238554684 podStartE2EDuration="2.238554684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:44:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:44:14.238086606 +0000 UTC m=+5136.267568903" watchObservedRunningTime="2026-11-03 02:44:14.238554684 +0000 UTC m=+5136.268036979" Nov 03 02:44:15.347207 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:44:15.347176 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7002-d660ba" Nov 03 02:44:15.463984 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:44:15.463895 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a5c4acd-245e-4b64-a146-dcc897eb8b8c-kube-api-access-pzsgg\" (UniqueName: \"kubernetes.io/projected/6a5c4acd-245e-4b64-a146-dcc897eb8b8c-kube-api-access-pzsgg\") pod \"6a5c4acd-245e-4b64-a146-dcc897eb8b8c\" (UID: \"6a5c4acd-245e-4b64-a146-dcc897eb8b8c\") " Nov 03 02:44:15.465903 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:44:15.465881 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a5c4acd-245e-4b64-a146-dcc897eb8b8c-kube-api-access-pzsgg" pod "6a5c4acd-245e-4b64-a146-dcc897eb8b8c" (UID: "6a5c4acd-245e-4b64-a146-dcc897eb8b8c"). InnerVolumeSpecName "kube-api-access-pzsgg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:44:15.564741 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:44:15.564705 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzsgg\" (UniqueName: \"kubernetes.io/projected/6a5c4acd-245e-4b64-a146-dcc897eb8b8c-kube-api-access-pzsgg\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:44:15.700358 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:44:15.700314 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7002-2c3c40"] Nov 03 02:44:15.706647 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:44:15.706612 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7002-2c3c40"] Nov 03 02:44:16.087885 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:44:16.087854 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a5c4acd-245e-4b64-a146-dcc897eb8b8c" path="/var/lib/kubelet/pods/6a5c4acd-245e-4b64-a146-dcc897eb8b8c/volumes" Nov 03 02:44:16.130137 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:44:16.130102 5437 generic.go:356] "Generic (PLEG): container finished" podID="6a5c4acd-245e-4b64-a146-dcc897eb8b8c" containerID="879bce6bf367af385675c69ef7bf8e26f124f5c698807dc3f2b80d883e623d1f" exitCode=137 Nov 03 02:44:16.130314 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:44:16.130162 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7002-d660ba" Nov 03 02:44:16.130314 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:44:16.130203 5437 scope.go:122] "RemoveContainer" containerID="879bce6bf367af385675c69ef7bf8e26f124f5c698807dc3f2b80d883e623d1f" Nov 03 02:44:16.130474 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:44:16.130402 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7002-2c3c40" podUID="04fc53e7-621c-4cf6-a890-2df029725785" containerName="test-curl-7002-2c3c40" containerID="cri-o://7121a63b872d89388c2fee718faf103d5d8e75ea9510a2abc1c2332176431ae3" gracePeriod=2 Nov 03 02:44:16.141056 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:44:16.141034 5437 scope.go:122] "RemoveContainer" containerID="879bce6bf367af385675c69ef7bf8e26f124f5c698807dc3f2b80d883e623d1f" Nov 03 02:44:16.141672 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:44:16.141644 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"879bce6bf367af385675c69ef7bf8e26f124f5c698807dc3f2b80d883e623d1f\": container with ID starting with 879bce6bf367af385675c69ef7bf8e26f124f5c698807dc3f2b80d883e623d1f not found: ID does not exist" containerID="879bce6bf367af385675c69ef7bf8e26f124f5c698807dc3f2b80d883e623d1f" Nov 03 02:44:16.141780 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:44:16.141675 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"879bce6bf367af385675c69ef7bf8e26f124f5c698807dc3f2b80d883e623d1f"} err="failed to get container status \"879bce6bf367af385675c69ef7bf8e26f124f5c698807dc3f2b80d883e623d1f\": rpc error: code = NotFound desc = could not find container \"879bce6bf367af385675c69ef7bf8e26f124f5c698807dc3f2b80d883e623d1f\": container with ID starting with 879bce6bf367af385675c69ef7bf8e26f124f5c698807dc3f2b80d883e623d1f not found: ID does not exist" Nov 03 02:44:17.904807 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:44:17.904714 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:44:18.351922 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:44:18.351897 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7002-2c3c40" Nov 03 02:44:18.495150 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:44:18.495066 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04fc53e7-621c-4cf6-a890-2df029725785-kube-api-access-bss4q\" (UniqueName: \"kubernetes.io/projected/04fc53e7-621c-4cf6-a890-2df029725785-kube-api-access-bss4q\") pod \"04fc53e7-621c-4cf6-a890-2df029725785\" (UID: \"04fc53e7-621c-4cf6-a890-2df029725785\") " Nov 03 02:44:18.497290 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:44:18.497263 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04fc53e7-621c-4cf6-a890-2df029725785-kube-api-access-bss4q" pod "04fc53e7-621c-4cf6-a890-2df029725785" (UID: "04fc53e7-621c-4cf6-a890-2df029725785"). InnerVolumeSpecName "kube-api-access-bss4q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:44:18.596683 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:44:18.596646 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bss4q\" (UniqueName: \"kubernetes.io/projected/04fc53e7-621c-4cf6-a890-2df029725785-kube-api-access-bss4q\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:44:19.147232 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:44:19.147196 5437 generic.go:356] "Generic (PLEG): container finished" podID="04fc53e7-621c-4cf6-a890-2df029725785" containerID="7121a63b872d89388c2fee718faf103d5d8e75ea9510a2abc1c2332176431ae3" exitCode=137 Nov 03 02:44:19.147721 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:44:19.147240 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7002-2c3c40" Nov 03 02:44:19.147721 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:44:19.147298 5437 scope.go:122] "RemoveContainer" containerID="7121a63b872d89388c2fee718faf103d5d8e75ea9510a2abc1c2332176431ae3" Nov 03 02:44:19.156087 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:44:19.156067 5437 scope.go:122] "RemoveContainer" containerID="7121a63b872d89388c2fee718faf103d5d8e75ea9510a2abc1c2332176431ae3" Nov 03 02:44:19.156440 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:44:19.156345 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7121a63b872d89388c2fee718faf103d5d8e75ea9510a2abc1c2332176431ae3\": container with ID starting with 7121a63b872d89388c2fee718faf103d5d8e75ea9510a2abc1c2332176431ae3 not found: ID does not exist" containerID="7121a63b872d89388c2fee718faf103d5d8e75ea9510a2abc1c2332176431ae3" Nov 03 02:44:19.156440 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:44:19.156391 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7121a63b872d89388c2fee718faf103d5d8e75ea9510a2abc1c2332176431ae3"} err="failed to get container status \"7121a63b872d89388c2fee718faf103d5d8e75ea9510a2abc1c2332176431ae3\": rpc error: code = NotFound desc = could not find container \"7121a63b872d89388c2fee718faf103d5d8e75ea9510a2abc1c2332176431ae3\": container with ID starting with 7121a63b872d89388c2fee718faf103d5d8e75ea9510a2abc1c2332176431ae3 not found: ID does not exist" Nov 03 02:44:19.208940 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:44:19.208905 5437 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7002-2c3c40\" is forbidden: User \"system:node:ip-10-0-136-16.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-16.us-west-2.compute.internal' and this object" podUID="04fc53e7-621c-4cf6-a890-2df029725785" pod="opendatahub/test-curl-7002-2c3c40" Nov 03 02:44:20.088403 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:44:20.088345 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04fc53e7-621c-4cf6-a890-2df029725785" path="/var/lib/kubelet/pods/04fc53e7-621c-4cf6-a890-2df029725785/volumes" Nov 03 02:44:22.678485 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:44:22.678444 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:44:27.470911 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:44:27.470872 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:44:32.271672 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:44:32.271636 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:44:37.091134 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:44:37.091090 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:44:41.931251 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:44:41.931209 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:44:44.034233 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:44:44.034195 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 02:44:46.701570 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:44:46.701523 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:44:51.489925 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:44:51.489884 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:44:56.287694 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:44:56.287624 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:45:00.166530 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:45:00.166490 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-a3dce7-29894565-m7fbp"] Nov 03 02:45:00.167025 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:45:00.166956 5437 cpu_manager.go:414] "containerMap: removing container" podUID="04fc53e7-621c-4cf6-a890-2df029725785" containerName="test-curl-7002-2c3c40" Nov 03 02:45:00.167025 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:45:00.166970 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04fc53e7-621c-4cf6-a890-2df029725785" containerName="test-curl-7002-2c3c40" Nov 03 02:45:00.167104 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:45:00.167056 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="04fc53e7-621c-4cf6-a890-2df029725785" containerName="test-curl-7002-2c3c40" Nov 03 02:45:00.170232 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:45:00.170213 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-a3dce7-29894565-m7fbp" Nov 03 02:45:00.255118 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:45:00.255082 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-a3dce7-29894565-m7fbp"] Nov 03 02:45:00.303439 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:45:00.303404 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbznp\" (UniqueName: \"kubernetes.io/projected/73a40f1f-042c-49ef-ab47-f62e7a1562e0-kube-api-access-sbznp\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-a3dce7-29894565-m7fbp\" (UID: \"73a40f1f-042c-49ef-ab47-f62e7a1562e0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-a3dce7-29894565-m7fbp" Nov 03 02:45:00.303670 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:45:00.303452 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73a40f1f-042c-49ef-ab47-f62e7a1562e0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-a3dce7-29894565-m7fbp\" (UID: \"73a40f1f-042c-49ef-ab47-f62e7a1562e0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-a3dce7-29894565-m7fbp" Nov 03 02:45:00.313859 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:45:00.313824 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fgwsl\"" Nov 03 02:45:00.404251 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:45:00.404206 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbznp\" (UniqueName: \"kubernetes.io/projected/73a40f1f-042c-49ef-ab47-f62e7a1562e0-kube-api-access-sbznp\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-a3dce7-29894565-m7fbp\" (UID: \"73a40f1f-042c-49ef-ab47-f62e7a1562e0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-a3dce7-29894565-m7fbp" Nov 03 02:45:00.404484 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:45:00.404279 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73a40f1f-042c-49ef-ab47-f62e7a1562e0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-a3dce7-29894565-m7fbp\" (UID: \"73a40f1f-042c-49ef-ab47-f62e7a1562e0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-a3dce7-29894565-m7fbp" Nov 03 02:45:00.404942 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:45:00.404919 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73a40f1f-042c-49ef-ab47-f62e7a1562e0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-a3dce7-29894565-m7fbp\" (UID: \"73a40f1f-042c-49ef-ab47-f62e7a1562e0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-a3dce7-29894565-m7fbp" Nov 03 02:45:00.467406 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:45:00.467316 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbznp\" (UniqueName: \"kubernetes.io/projected/73a40f1f-042c-49ef-ab47-f62e7a1562e0-kube-api-access-sbznp\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-a3dce7-29894565-m7fbp\" (UID: \"73a40f1f-042c-49ef-ab47-f62e7a1562e0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-a3dce7-29894565-m7fbp" Nov 03 02:45:00.483118 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:45:00.483087 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-a3dce7-29894565-m7fbp" Nov 03 02:45:00.854034 ip-10-0-136-16 kubenswrapper[5437]: W1103 02:45:00.853999 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73a40f1f_042c_49ef_ab47_f62e7a1562e0.slice/crio-789fb2ee07263ada91099bf3460252bb054b09a2f96ca7d8d6555e1489e052f7 WatchSource:0}: Error finding container 789fb2ee07263ada91099bf3460252bb054b09a2f96ca7d8d6555e1489e052f7: Status 404 returned error can't find the container with id 789fb2ee07263ada91099bf3460252bb054b09a2f96ca7d8d6555e1489e052f7 Nov 03 02:45:00.855973 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:45:00.855951 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-a3dce7-29894565-m7fbp"] Nov 03 02:45:01.084102 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:45:01.084059 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:45:01.367694 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:45:01.367650 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-a3dce7-29894565-m7fbp" event={"ID":"73a40f1f-042c-49ef-ab47-f62e7a1562e0","Type":"ContainerStarted","Data":"0352ec58f2cc3cb6d591826d979a4fd14e49c236eeb7b1337a8d26914e6268eb"} Nov 03 02:45:01.368097 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:45:01.367711 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-a3dce7-29894565-m7fbp" event={"ID":"73a40f1f-042c-49ef-ab47-f62e7a1562e0","Type":"ContainerStarted","Data":"789fb2ee07263ada91099bf3460252bb054b09a2f96ca7d8d6555e1489e052f7"} Nov 03 02:45:01.491456 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:45:01.491325 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-a3dce7-29894565-m7fbp" podStartSLOduration=1.491306641 podStartE2EDuration="1.491306641s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:45:01.490011954 +0000 UTC m=+5183.519494249" watchObservedRunningTime="2026-11-03 02:45:01.491306641 +0000 UTC m=+5183.520788937" Nov 03 02:45:02.278416 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:45:02.278375 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-a3dce7/sub-test-model-a3dce7-kserve-78f7fc75f-pkntl"] Nov 03 02:45:02.345184 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:45:02.345133 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:45:02.373684 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:45:02.373649 5437 generic.go:356] "Generic (PLEG): container finished" podID="73a40f1f-042c-49ef-ab47-f62e7a1562e0" containerID="0352ec58f2cc3cb6d591826d979a4fd14e49c236eeb7b1337a8d26914e6268eb" exitCode=60 Nov 03 02:45:02.374080 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:45:02.373737 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-a3dce7-29894565-m7fbp" event={"ID":"73a40f1f-042c-49ef-ab47-f62e7a1562e0","Type":"ContainerDied","Data":"0352ec58f2cc3cb6d591826d979a4fd14e49c236eeb7b1337a8d26914e6268eb"} Nov 03 02:45:02.374080 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:45:02.373946 5437 scope.go:122] "RemoveContainer" containerID="0352ec58f2cc3cb6d591826d979a4fd14e49c236eeb7b1337a8d26914e6268eb" Nov 03 02:45:03.381795 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:45:03.381757 5437 generic.go:356] "Generic (PLEG): container finished" podID="73a40f1f-042c-49ef-ab47-f62e7a1562e0" containerID="fa3462e1511dd42bea6b26916d9f0253f12214ea359a3f1f0c73a10ed123d956" exitCode=60 Nov 03 02:45:03.382352 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:45:03.381859 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-a3dce7-29894565-m7fbp" event={"ID":"73a40f1f-042c-49ef-ab47-f62e7a1562e0","Type":"ContainerDied","Data":"fa3462e1511dd42bea6b26916d9f0253f12214ea359a3f1f0c73a10ed123d956"} Nov 03 02:45:03.382352 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:45:03.381919 5437 scope.go:122] "RemoveContainer" containerID="0352ec58f2cc3cb6d591826d979a4fd14e49c236eeb7b1337a8d26914e6268eb" Nov 03 02:45:03.382352 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:45:03.382141 5437 scope.go:122] "RemoveContainer" containerID="fa3462e1511dd42bea6b26916d9f0253f12214ea359a3f1f0c73a10ed123d956" Nov 03 02:45:03.382550 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:45:03.382461 5437 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-a3dce7-29894565-m7fbp_odh-ai-gateway-infra(73a40f1f-042c-49ef-ab47-f62e7a1562e0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-a3dce7-29894565-m7fbp" podUID="73a40f1f-042c-49ef-ab47-f62e7a1562e0" Nov 03 02:45:04.389406 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:45:04.389376 5437 scope.go:122] "RemoveContainer" containerID="fa3462e1511dd42bea6b26916d9f0253f12214ea359a3f1f0c73a10ed123d956" Nov 03 02:45:04.389829 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:45:04.389641 5437 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-a3dce7-29894565-m7fbp_odh-ai-gateway-infra(73a40f1f-042c-49ef-ab47-f62e7a1562e0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-a3dce7-29894565-m7fbp" podUID="73a40f1f-042c-49ef-ab47-f62e7a1562e0" Nov 03 02:45:05.305383 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:45:05.305332 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:45:05.984140 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:45:05.984098 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:45:10.298195 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:45:10.298152 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:45:10.888863 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:45:10.888817 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:45:14.034192 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:45:14.034141 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 02:45:15.703125 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:45:15.703083 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:45:17.082253 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:45:17.082217 5437 scope.go:122] "RemoveContainer" containerID="fa3462e1511dd42bea6b26916d9f0253f12214ea359a3f1f0c73a10ed123d956" Nov 03 02:45:17.298810 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:45:17.298766 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w0-a3dce7/sub-test-model-a3dce7-kserve-78f7fc75f-pkntl" podUID="2cac18c4-e538-4c4b-85cf-de232f64de18" containerName="main" containerID="cri-o://dde6a8d919e274d95a2526310d31569aee9e706cfc4f2e80fa62b0b399a7ed16" gracePeriod=45 Nov 03 02:45:17.457426 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:45:17.457394 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-a3dce7/sub-test-model-a3dce7-kserve-78f7fc75f-pkntl" event={"ID":"2cac18c4-e538-4c4b-85cf-de232f64de18","Type":"ContainerDied","Data":"dde6a8d919e274d95a2526310d31569aee9e706cfc4f2e80fa62b0b399a7ed16"} Nov 03 02:45:17.457626 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:45:17.457359 5437 generic.go:356] "Generic (PLEG): container finished" podID="2cac18c4-e538-4c4b-85cf-de232f64de18" containerID="dde6a8d919e274d95a2526310d31569aee9e706cfc4f2e80fa62b0b399a7ed16" exitCode=0 Nov 03 02:45:17.459833 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:45:17.459801 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-a3dce7-29894565-m7fbp" event={"ID":"73a40f1f-042c-49ef-ab47-f62e7a1562e0","Type":"ContainerStarted","Data":"c4d0a29e27bd23e4d4c59dd0544d2084a07890354153454dcc1eb40a27942b8d"} Nov 03 02:45:17.561012 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:45:17.560984 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-a3dce7/sub-test-model-a3dce7-kserve-78f7fc75f-pkntl" Nov 03 02:45:17.675750 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:45:17.675669 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cac18c4-e538-4c4b-85cf-de232f64de18-kube-api-access-dn5lb\" (UniqueName: \"kubernetes.io/projected/2cac18c4-e538-4c4b-85cf-de232f64de18-kube-api-access-dn5lb\") pod \"2cac18c4-e538-4c4b-85cf-de232f64de18\" (UID: \"2cac18c4-e538-4c4b-85cf-de232f64de18\") " Nov 03 02:45:17.675750 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:45:17.675716 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2cac18c4-e538-4c4b-85cf-de232f64de18-tls-certs\" (UniqueName: \"kubernetes.io/secret/2cac18c4-e538-4c4b-85cf-de232f64de18-tls-certs\") pod \"2cac18c4-e538-4c4b-85cf-de232f64de18\" (UID: \"2cac18c4-e538-4c4b-85cf-de232f64de18\") " Nov 03 02:45:17.675956 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:45:17.675759 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cac18c4-e538-4c4b-85cf-de232f64de18-home\" (UniqueName: \"kubernetes.io/empty-dir/2cac18c4-e538-4c4b-85cf-de232f64de18-home\") pod \"2cac18c4-e538-4c4b-85cf-de232f64de18\" (UID: \"2cac18c4-e538-4c4b-85cf-de232f64de18\") " Nov 03 02:45:17.675956 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:45:17.675866 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cac18c4-e538-4c4b-85cf-de232f64de18-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cac18c4-e538-4c4b-85cf-de232f64de18-model-cache\") pod \"2cac18c4-e538-4c4b-85cf-de232f64de18\" (UID: \"2cac18c4-e538-4c4b-85cf-de232f64de18\") " Nov 03 02:45:17.675956 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:45:17.675904 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cac18c4-e538-4c4b-85cf-de232f64de18-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2cac18c4-e538-4c4b-85cf-de232f64de18-dshm\") pod \"2cac18c4-e538-4c4b-85cf-de232f64de18\" (UID: \"2cac18c4-e538-4c4b-85cf-de232f64de18\") " Nov 03 02:45:17.676130 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:45:17.675953 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cac18c4-e538-4c4b-85cf-de232f64de18-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cac18c4-e538-4c4b-85cf-de232f64de18-tmp-dir\") pod \"2cac18c4-e538-4c4b-85cf-de232f64de18\" (UID: \"2cac18c4-e538-4c4b-85cf-de232f64de18\") " Nov 03 02:45:17.676192 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:45:17.676099 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cac18c4-e538-4c4b-85cf-de232f64de18-home" pod "2cac18c4-e538-4c4b-85cf-de232f64de18" (UID: "2cac18c4-e538-4c4b-85cf-de232f64de18"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:45:17.676255 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:45:17.676183 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cac18c4-e538-4c4b-85cf-de232f64de18-model-cache" pod "2cac18c4-e538-4c4b-85cf-de232f64de18" (UID: "2cac18c4-e538-4c4b-85cf-de232f64de18"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:45:17.676355 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:45:17.676260 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cac18c4-e538-4c4b-85cf-de232f64de18-tmp-dir" pod "2cac18c4-e538-4c4b-85cf-de232f64de18" (UID: "2cac18c4-e538-4c4b-85cf-de232f64de18"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:45:17.676447 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:45:17.676391 5437 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2cac18c4-e538-4c4b-85cf-de232f64de18-home\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:45:17.676447 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:45:17.676407 5437 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cac18c4-e538-4c4b-85cf-de232f64de18-model-cache\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:45:17.676447 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:45:17.676417 5437 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cac18c4-e538-4c4b-85cf-de232f64de18-tmp-dir\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:45:17.678049 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:45:17.678027 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cac18c4-e538-4c4b-85cf-de232f64de18-kube-api-access-dn5lb" pod "2cac18c4-e538-4c4b-85cf-de232f64de18" (UID: "2cac18c4-e538-4c4b-85cf-de232f64de18"). InnerVolumeSpecName "kube-api-access-dn5lb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:45:17.678338 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:45:17.678319 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cac18c4-e538-4c4b-85cf-de232f64de18-dshm" pod "2cac18c4-e538-4c4b-85cf-de232f64de18" (UID: "2cac18c4-e538-4c4b-85cf-de232f64de18"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:45:17.678399 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:45:17.678334 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2cac18c4-e538-4c4b-85cf-de232f64de18-tls-certs" pod "2cac18c4-e538-4c4b-85cf-de232f64de18" (UID: "2cac18c4-e538-4c4b-85cf-de232f64de18"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:45:17.777889 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:45:17.777855 5437 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2cac18c4-e538-4c4b-85cf-de232f64de18-dshm\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:45:17.777889 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:45:17.777885 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dn5lb\" (UniqueName: \"kubernetes.io/projected/2cac18c4-e538-4c4b-85cf-de232f64de18-kube-api-access-dn5lb\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:45:17.777889 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:45:17.777895 5437 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2cac18c4-e538-4c4b-85cf-de232f64de18-tls-certs\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:45:18.466570 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:45:18.466491 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-a3dce7/sub-test-model-a3dce7-kserve-78f7fc75f-pkntl" Nov 03 02:45:18.467042 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:45:18.466487 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-a3dce7/sub-test-model-a3dce7-kserve-78f7fc75f-pkntl" event={"ID":"2cac18c4-e538-4c4b-85cf-de232f64de18","Type":"ContainerDied","Data":"f2dca03c88879dad366f764e202b42cee47c8b7987b4706561589f831c74f41e"} Nov 03 02:45:18.467042 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:45:18.466617 5437 scope.go:122] "RemoveContainer" containerID="dde6a8d919e274d95a2526310d31569aee9e706cfc4f2e80fa62b0b399a7ed16" Nov 03 02:45:18.468956 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:45:18.468927 5437 generic.go:356] "Generic (PLEG): container finished" podID="73a40f1f-042c-49ef-ab47-f62e7a1562e0" containerID="c4d0a29e27bd23e4d4c59dd0544d2084a07890354153454dcc1eb40a27942b8d" exitCode=60 Nov 03 02:45:18.469102 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:45:18.468973 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-a3dce7-29894565-m7fbp" event={"ID":"73a40f1f-042c-49ef-ab47-f62e7a1562e0","Type":"ContainerDied","Data":"c4d0a29e27bd23e4d4c59dd0544d2084a07890354153454dcc1eb40a27942b8d"} Nov 03 02:45:18.469333 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:45:18.469318 5437 scope.go:122] "RemoveContainer" containerID="c4d0a29e27bd23e4d4c59dd0544d2084a07890354153454dcc1eb40a27942b8d" Nov 03 02:45:18.469629 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:45:18.469602 5437 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-a3dce7-29894565-m7fbp_odh-ai-gateway-infra(73a40f1f-042c-49ef-ab47-f62e7a1562e0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-a3dce7-29894565-m7fbp" podUID="73a40f1f-042c-49ef-ab47-f62e7a1562e0" Nov 03 02:45:18.475699 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:45:18.475676 5437 scope.go:122] "RemoveContainer" containerID="fa3462e1511dd42bea6b26916d9f0253f12214ea359a3f1f0c73a10ed123d956" Nov 03 02:45:18.591984 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:45:18.591950 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-a3dce7-29894565-m7fbp"] Nov 03 02:45:18.753116 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:45:18.753042 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-a3dce7/sub-test-model-a3dce7-kserve-78f7fc75f-pkntl"] Nov 03 02:45:18.757023 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:45:18.756993 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w0-a3dce7/sub-test-model-a3dce7-kserve-78f7fc75f-pkntl"] Nov 03 02:45:19.617850 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:45:19.617826 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-a3dce7-29894565-m7fbp" Nov 03 02:45:19.697202 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:45:19.697171 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73a40f1f-042c-49ef-ab47-f62e7a1562e0-kube-api-access-sbznp\" (UniqueName: \"kubernetes.io/projected/73a40f1f-042c-49ef-ab47-f62e7a1562e0-kube-api-access-sbznp\") pod \"73a40f1f-042c-49ef-ab47-f62e7a1562e0\" (UID: \"73a40f1f-042c-49ef-ab47-f62e7a1562e0\") " Nov 03 02:45:19.697424 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:45:19.697324 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73a40f1f-042c-49ef-ab47-f62e7a1562e0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73a40f1f-042c-49ef-ab47-f62e7a1562e0-maas-api-service-ca\") pod \"73a40f1f-042c-49ef-ab47-f62e7a1562e0\" (UID: \"73a40f1f-042c-49ef-ab47-f62e7a1562e0\") " Nov 03 02:45:19.697709 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:45:19.697680 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73a40f1f-042c-49ef-ab47-f62e7a1562e0-maas-api-service-ca" pod "73a40f1f-042c-49ef-ab47-f62e7a1562e0" (UID: "73a40f1f-042c-49ef-ab47-f62e7a1562e0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:45:19.699289 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:45:19.699267 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73a40f1f-042c-49ef-ab47-f62e7a1562e0-kube-api-access-sbznp" pod "73a40f1f-042c-49ef-ab47-f62e7a1562e0" (UID: "73a40f1f-042c-49ef-ab47-f62e7a1562e0"). InnerVolumeSpecName "kube-api-access-sbznp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:45:19.798331 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:45:19.798297 5437 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73a40f1f-042c-49ef-ab47-f62e7a1562e0-maas-api-service-ca\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:45:19.798331 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:45:19.798331 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbznp\" (UniqueName: \"kubernetes.io/projected/73a40f1f-042c-49ef-ab47-f62e7a1562e0-kube-api-access-sbznp\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:45:20.088619 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:45:20.088542 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2cac18c4-e538-4c4b-85cf-de232f64de18" path="/var/lib/kubelet/pods/2cac18c4-e538-4c4b-85cf-de232f64de18/volumes" Nov 03 02:45:20.487174 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:45:20.487094 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-a3dce7-29894565-m7fbp" event={"ID":"73a40f1f-042c-49ef-ab47-f62e7a1562e0","Type":"ContainerDied","Data":"789fb2ee07263ada91099bf3460252bb054b09a2f96ca7d8d6555e1489e052f7"} Nov 03 02:45:20.487174 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:45:20.487158 5437 scope.go:122] "RemoveContainer" containerID="c4d0a29e27bd23e4d4c59dd0544d2084a07890354153454dcc1eb40a27942b8d" Nov 03 02:45:20.487421 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:45:20.487106 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-a3dce7-29894565-m7fbp" Nov 03 02:45:20.494106 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:45:20.494066 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:45:20.656254 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:45:20.656216 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-a3dce7-29894565-m7fbp"] Nov 03 02:45:20.660145 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:45:20.660117 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-a3dce7-29894565-m7fbp"] Nov 03 02:45:22.087944 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:45:22.087909 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73a40f1f-042c-49ef-ab47-f62e7a1562e0" path="/var/lib/kubelet/pods/73a40f1f-042c-49ef-ab47-f62e7a1562e0/volumes" Nov 03 02:45:25.282592 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:45:25.282553 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:45:27.517291 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:45:27.517254 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-f0391a/sub-test-model-f0391a-kserve-dd566f79c-sqxqq"] Nov 03 02:45:27.592099 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:45:27.592058 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:45:28.080540 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:45:28.080499 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:45:29.997938 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:45:29.997903 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:45:33.516780 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:45:33.516740 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:45:34.204801 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:45:34.204765 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:45:34.801417 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:45:34.801349 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:45:35.082673 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:45:35.082597 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:45:36.225783 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:45:36.225749 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:45:40.756990 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:45:40.756950 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:45:41.014416 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:45:41.014299 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:45:41.219656 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:45:41.219612 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:45:42.275670 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:45:42.275601 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:45:42.539026 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:45:42.538939 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w0-f0391a/sub-test-model-f0391a-kserve-dd566f79c-sqxqq" podUID="f4467722-3cdf-4e97-b559-2e5e5b0b62db" containerName="main" containerID="cri-o://46778fc78ed4eb4f02f6c0c1ae18bdd45c4bb543440778be2f44079d5a51a828" gracePeriod=45 Nov 03 02:45:42.658398 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:45:42.658347 5437 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w0-f0391a/sub-test-model-f0391a-kserve-dd566f79c-sqxqq" podUID="f4467722-3cdf-4e97-b559-2e5e5b0b62db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.189:8000/ready\": dial tcp 10.134.0.189:8000: connect: connection refused" Nov 03 02:45:42.787896 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:45:42.787866 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-f0391a/sub-test-model-f0391a-kserve-dd566f79c-sqxqq" Nov 03 02:45:42.914998 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:45:42.914961 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:45:42.927934 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:45:42.927900 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4467722-3cdf-4e97-b559-2e5e5b0b62db-home\" (UniqueName: \"kubernetes.io/empty-dir/f4467722-3cdf-4e97-b559-2e5e5b0b62db-home\") pod \"f4467722-3cdf-4e97-b559-2e5e5b0b62db\" (UID: \"f4467722-3cdf-4e97-b559-2e5e5b0b62db\") " Nov 03 02:45:42.928108 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:45:42.927970 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4467722-3cdf-4e97-b559-2e5e5b0b62db-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4467722-3cdf-4e97-b559-2e5e5b0b62db-tmp-dir\") pod \"f4467722-3cdf-4e97-b559-2e5e5b0b62db\" (UID: \"f4467722-3cdf-4e97-b559-2e5e5b0b62db\") " Nov 03 02:45:42.928108 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:45:42.927999 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4467722-3cdf-4e97-b559-2e5e5b0b62db-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4467722-3cdf-4e97-b559-2e5e5b0b62db-dshm\") pod \"f4467722-3cdf-4e97-b559-2e5e5b0b62db\" (UID: \"f4467722-3cdf-4e97-b559-2e5e5b0b62db\") " Nov 03 02:45:42.928108 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:45:42.928055 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4467722-3cdf-4e97-b559-2e5e5b0b62db-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4467722-3cdf-4e97-b559-2e5e5b0b62db-model-cache\") pod \"f4467722-3cdf-4e97-b559-2e5e5b0b62db\" (UID: \"f4467722-3cdf-4e97-b559-2e5e5b0b62db\") " Nov 03 02:45:42.928272 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:45:42.928109 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4467722-3cdf-4e97-b559-2e5e5b0b62db-kube-api-access-2ks6z\" (UniqueName: \"kubernetes.io/projected/f4467722-3cdf-4e97-b559-2e5e5b0b62db-kube-api-access-2ks6z\") pod \"f4467722-3cdf-4e97-b559-2e5e5b0b62db\" (UID: \"f4467722-3cdf-4e97-b559-2e5e5b0b62db\") " Nov 03 02:45:42.928272 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:45:42.928137 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4467722-3cdf-4e97-b559-2e5e5b0b62db-tls-certs\" (UniqueName: \"kubernetes.io/secret/f4467722-3cdf-4e97-b559-2e5e5b0b62db-tls-certs\") pod \"f4467722-3cdf-4e97-b559-2e5e5b0b62db\" (UID: \"f4467722-3cdf-4e97-b559-2e5e5b0b62db\") " Nov 03 02:45:42.928272 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:45:42.928215 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4467722-3cdf-4e97-b559-2e5e5b0b62db-home" pod "f4467722-3cdf-4e97-b559-2e5e5b0b62db" (UID: "f4467722-3cdf-4e97-b559-2e5e5b0b62db"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:45:42.928441 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:45:42.928277 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4467722-3cdf-4e97-b559-2e5e5b0b62db-tmp-dir" pod "f4467722-3cdf-4e97-b559-2e5e5b0b62db" (UID: "f4467722-3cdf-4e97-b559-2e5e5b0b62db"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:45:42.928441 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:45:42.928334 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4467722-3cdf-4e97-b559-2e5e5b0b62db-model-cache" pod "f4467722-3cdf-4e97-b559-2e5e5b0b62db" (UID: "f4467722-3cdf-4e97-b559-2e5e5b0b62db"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:45:42.928529 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:45:42.928514 5437 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4467722-3cdf-4e97-b559-2e5e5b0b62db-home\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:45:42.928594 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:45:42.928551 5437 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4467722-3cdf-4e97-b559-2e5e5b0b62db-tmp-dir\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:45:42.928594 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:45:42.928561 5437 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4467722-3cdf-4e97-b559-2e5e5b0b62db-model-cache\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:45:42.930391 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:45:42.930346 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4467722-3cdf-4e97-b559-2e5e5b0b62db-tls-certs" pod "f4467722-3cdf-4e97-b559-2e5e5b0b62db" (UID: "f4467722-3cdf-4e97-b559-2e5e5b0b62db"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:45:42.930510 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:45:42.930354 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4467722-3cdf-4e97-b559-2e5e5b0b62db-dshm" pod "f4467722-3cdf-4e97-b559-2e5e5b0b62db" (UID: "f4467722-3cdf-4e97-b559-2e5e5b0b62db"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:45:42.930627 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:45:42.930607 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4467722-3cdf-4e97-b559-2e5e5b0b62db-kube-api-access-2ks6z" pod "f4467722-3cdf-4e97-b559-2e5e5b0b62db" (UID: "f4467722-3cdf-4e97-b559-2e5e5b0b62db"). InnerVolumeSpecName "kube-api-access-2ks6z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:45:43.029455 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:45:43.029419 5437 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4467722-3cdf-4e97-b559-2e5e5b0b62db-dshm\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:45:43.029455 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:45:43.029451 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ks6z\" (UniqueName: \"kubernetes.io/projected/f4467722-3cdf-4e97-b559-2e5e5b0b62db-kube-api-access-2ks6z\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:45:43.029455 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:45:43.029462 5437 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4467722-3cdf-4e97-b559-2e5e5b0b62db-tls-certs\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:45:43.612502 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:45:43.612450 5437 generic.go:356] "Generic (PLEG): container finished" podID="f4467722-3cdf-4e97-b559-2e5e5b0b62db" containerID="46778fc78ed4eb4f02f6c0c1ae18bdd45c4bb543440778be2f44079d5a51a828" exitCode=0 Nov 03 02:45:43.612998 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:45:43.612593 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-f0391a/sub-test-model-f0391a-kserve-dd566f79c-sqxqq" event={"ID":"f4467722-3cdf-4e97-b559-2e5e5b0b62db","Type":"ContainerDied","Data":"46778fc78ed4eb4f02f6c0c1ae18bdd45c4bb543440778be2f44079d5a51a828"} Nov 03 02:45:43.612998 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:45:43.612640 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-f0391a/sub-test-model-f0391a-kserve-dd566f79c-sqxqq" Nov 03 02:45:43.612998 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:45:43.612666 5437 scope.go:122] "RemoveContainer" containerID="46778fc78ed4eb4f02f6c0c1ae18bdd45c4bb543440778be2f44079d5a51a828" Nov 03 02:45:43.612998 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:45:43.612656 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-f0391a/sub-test-model-f0391a-kserve-dd566f79c-sqxqq" event={"ID":"f4467722-3cdf-4e97-b559-2e5e5b0b62db","Type":"ContainerDied","Data":"a3b8b06caf8cf60981210b7c663016ed557993fdce98cc433c18a40ea84a74a0"} Nov 03 02:45:43.621969 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:45:43.621950 5437 scope.go:122] "RemoveContainer" containerID="46778fc78ed4eb4f02f6c0c1ae18bdd45c4bb543440778be2f44079d5a51a828" Nov 03 02:45:43.622244 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:45:43.622225 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46778fc78ed4eb4f02f6c0c1ae18bdd45c4bb543440778be2f44079d5a51a828\": container with ID starting with 46778fc78ed4eb4f02f6c0c1ae18bdd45c4bb543440778be2f44079d5a51a828 not found: ID does not exist" containerID="46778fc78ed4eb4f02f6c0c1ae18bdd45c4bb543440778be2f44079d5a51a828" Nov 03 02:45:43.622284 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:45:43.622253 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46778fc78ed4eb4f02f6c0c1ae18bdd45c4bb543440778be2f44079d5a51a828"} err="failed to get container status \"46778fc78ed4eb4f02f6c0c1ae18bdd45c4bb543440778be2f44079d5a51a828\": rpc error: code = NotFound desc = could not find container \"46778fc78ed4eb4f02f6c0c1ae18bdd45c4bb543440778be2f44079d5a51a828\": container with ID starting with 46778fc78ed4eb4f02f6c0c1ae18bdd45c4bb543440778be2f44079d5a51a828 not found: ID does not exist" Nov 03 02:45:43.733784 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:45:43.733747 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:45:43.781424 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:45:43.781386 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-f0391a/sub-test-model-f0391a-kserve-dd566f79c-sqxqq"] Nov 03 02:45:43.833100 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:45:43.833064 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w0-f0391a/sub-test-model-f0391a-kserve-dd566f79c-sqxqq"] Nov 03 02:45:44.034819 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:45:44.034731 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 02:45:44.087115 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:45:44.087072 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4467722-3cdf-4e97-b559-2e5e5b0b62db" path="/var/lib/kubelet/pods/f4467722-3cdf-4e97-b559-2e5e5b0b62db/volumes" Nov 03 02:45:46.224967 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:45:46.224930 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:45:49.678196 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:45:49.678108 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:45:50.503106 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:45:50.503065 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:45:50.759511 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:45:50.759412 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:45:52.489049 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:45:52.489011 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:45:55.631882 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:45:55.631842 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:45:55.844231 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:45:55.844191 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:45:57.235589 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:45:57.235551 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:45:57.486962 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:45:57.486879 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:45:57.734252 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:45:57.734210 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:45:58.221623 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:45:58.221588 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:46:02.679863 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:02.679818 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:46:03.271358 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:03.271326 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:46:04.660406 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:04.660343 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:46:04.861648 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:04.861608 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:46:05.294809 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:05.294768 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:46:05.646272 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:05.646240 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:46:07.470605 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:07.470570 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:46:10.615737 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:10.615702 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:46:11.407500 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:11.407444 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:46:11.597823 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:11.597777 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-b6fb9b-8b66f7f77-5xglx"] Nov 03 02:46:11.705360 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:11.705276 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-b6fb9b-666f59b59f-7d5tv"] Nov 03 02:46:11.995588 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:11.995461 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:46:12.200229 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:12.200193 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:46:14.035113 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:46:14.035074 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 02:46:16.599058 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:16.598976 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-b6fb9b-8b66f7f77-5xglx" podUID="0c2adf4b-1195-4937-b7d1-ea164abf40ab" containerName="payload-processing" containerID="cri-o://d6e3091dbb32da201a31c0c68e925ce035764de85860a377f0d18c5056e1d8c5" gracePeriod=25 Nov 03 02:46:16.705727 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:16.705659 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b6fb9b-666f59b59f-7d5tv" podUID="00f0ee4a-6616-454f-8de4-b39189d45442" containerName="payload-pre-processing" containerID="cri-o://8d8aaa5019ad072a5d9e32b8c7f6dbd0a4d19b4f0b384ce6f58a656bbfa29ff7" gracePeriod=25 Nov 03 02:46:16.784958 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:16.784924 5437 generic.go:356] "Generic (PLEG): container finished" podID="0c2adf4b-1195-4937-b7d1-ea164abf40ab" containerID="d6e3091dbb32da201a31c0c68e925ce035764de85860a377f0d18c5056e1d8c5" exitCode=0 Nov 03 02:46:16.785143 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:16.784996 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-b6fb9b-8b66f7f77-5xglx" event={"ID":"0c2adf4b-1195-4937-b7d1-ea164abf40ab","Type":"ContainerDied","Data":"d6e3091dbb32da201a31c0c68e925ce035764de85860a377f0d18c5056e1d8c5"} Nov 03 02:46:16.857118 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:16.857044 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-b6fb9b-8b66f7f77-5xglx" Nov 03 02:46:16.963177 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:16.963149 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b6fb9b-666f59b59f-7d5tv" Nov 03 02:46:16.968928 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:16.968901 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c2adf4b-1195-4937-b7d1-ea164abf40ab-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c2adf4b-1195-4937-b7d1-ea164abf40ab-plugins-config-volume\") pod \"0c2adf4b-1195-4937-b7d1-ea164abf40ab\" (UID: \"0c2adf4b-1195-4937-b7d1-ea164abf40ab\") " Nov 03 02:46:16.969090 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:16.969040 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c2adf4b-1195-4937-b7d1-ea164abf40ab-kube-api-access-llld6\" (UniqueName: \"kubernetes.io/projected/0c2adf4b-1195-4937-b7d1-ea164abf40ab-kube-api-access-llld6\") pod \"0c2adf4b-1195-4937-b7d1-ea164abf40ab\" (UID: \"0c2adf4b-1195-4937-b7d1-ea164abf40ab\") " Nov 03 02:46:16.969280 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:16.969254 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c2adf4b-1195-4937-b7d1-ea164abf40ab-plugins-config-volume" pod "0c2adf4b-1195-4937-b7d1-ea164abf40ab" (UID: "0c2adf4b-1195-4937-b7d1-ea164abf40ab"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:46:16.971097 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:16.971077 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c2adf4b-1195-4937-b7d1-ea164abf40ab-kube-api-access-llld6" pod "0c2adf4b-1195-4937-b7d1-ea164abf40ab" (UID: "0c2adf4b-1195-4937-b7d1-ea164abf40ab"). InnerVolumeSpecName "kube-api-access-llld6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:46:17.070191 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:17.070149 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00f0ee4a-6616-454f-8de4-b39189d45442-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00f0ee4a-6616-454f-8de4-b39189d45442-plugins-config-volume\") pod \"00f0ee4a-6616-454f-8de4-b39189d45442\" (UID: \"00f0ee4a-6616-454f-8de4-b39189d45442\") " Nov 03 02:46:17.070420 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:17.070296 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00f0ee4a-6616-454f-8de4-b39189d45442-kube-api-access-xxkwb\" (UniqueName: \"kubernetes.io/projected/00f0ee4a-6616-454f-8de4-b39189d45442-kube-api-access-xxkwb\") pod \"00f0ee4a-6616-454f-8de4-b39189d45442\" (UID: \"00f0ee4a-6616-454f-8de4-b39189d45442\") " Nov 03 02:46:17.070516 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:17.070501 5437 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c2adf4b-1195-4937-b7d1-ea164abf40ab-plugins-config-volume\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:46:17.070573 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:17.070518 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-llld6\" (UniqueName: \"kubernetes.io/projected/0c2adf4b-1195-4937-b7d1-ea164abf40ab-kube-api-access-llld6\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:46:17.070619 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:17.070570 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00f0ee4a-6616-454f-8de4-b39189d45442-plugins-config-volume" pod "00f0ee4a-6616-454f-8de4-b39189d45442" (UID: "00f0ee4a-6616-454f-8de4-b39189d45442"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:46:17.072491 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:17.072462 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00f0ee4a-6616-454f-8de4-b39189d45442-kube-api-access-xxkwb" pod "00f0ee4a-6616-454f-8de4-b39189d45442" (UID: "00f0ee4a-6616-454f-8de4-b39189d45442"). InnerVolumeSpecName "kube-api-access-xxkwb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:46:17.171889 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:17.171789 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxkwb\" (UniqueName: \"kubernetes.io/projected/00f0ee4a-6616-454f-8de4-b39189d45442-kube-api-access-xxkwb\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:46:17.171889 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:17.171834 5437 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00f0ee4a-6616-454f-8de4-b39189d45442-plugins-config-volume\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:46:17.285151 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:17.285114 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:46:17.791562 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:17.791529 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-b6fb9b-8b66f7f77-5xglx" Nov 03 02:46:17.792059 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:17.791564 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-b6fb9b-8b66f7f77-5xglx" event={"ID":"0c2adf4b-1195-4937-b7d1-ea164abf40ab","Type":"ContainerDied","Data":"14db8b36c8647b14086b7fe28acd4e7aca87ceca9d6f956968d47eabbff0c311"} Nov 03 02:46:17.792059 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:17.791639 5437 scope.go:122] "RemoveContainer" containerID="d6e3091dbb32da201a31c0c68e925ce035764de85860a377f0d18c5056e1d8c5" Nov 03 02:46:17.793011 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:17.792985 5437 generic.go:356] "Generic (PLEG): container finished" podID="00f0ee4a-6616-454f-8de4-b39189d45442" containerID="8d8aaa5019ad072a5d9e32b8c7f6dbd0a4d19b4f0b384ce6f58a656bbfa29ff7" exitCode=0 Nov 03 02:46:17.793097 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:17.793022 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b6fb9b-666f59b59f-7d5tv" Nov 03 02:46:17.793097 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:17.793031 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b6fb9b-666f59b59f-7d5tv" event={"ID":"00f0ee4a-6616-454f-8de4-b39189d45442","Type":"ContainerDied","Data":"8d8aaa5019ad072a5d9e32b8c7f6dbd0a4d19b4f0b384ce6f58a656bbfa29ff7"} Nov 03 02:46:17.793097 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:17.793088 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b6fb9b-666f59b59f-7d5tv" event={"ID":"00f0ee4a-6616-454f-8de4-b39189d45442","Type":"ContainerDied","Data":"e08b7bdd9c1ebd370ca3ac50366c667edcb7959fe167fffca78992aa7fdbef1b"} Nov 03 02:46:17.802332 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:17.802310 5437 scope.go:122] "RemoveContainer" containerID="8d8aaa5019ad072a5d9e32b8c7f6dbd0a4d19b4f0b384ce6f58a656bbfa29ff7" Nov 03 02:46:17.810792 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:17.810771 5437 scope.go:122] "RemoveContainer" containerID="8d8aaa5019ad072a5d9e32b8c7f6dbd0a4d19b4f0b384ce6f58a656bbfa29ff7" Nov 03 02:46:17.811070 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:46:17.811050 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d8aaa5019ad072a5d9e32b8c7f6dbd0a4d19b4f0b384ce6f58a656bbfa29ff7\": container with ID starting with 8d8aaa5019ad072a5d9e32b8c7f6dbd0a4d19b4f0b384ce6f58a656bbfa29ff7 not found: ID does not exist" containerID="8d8aaa5019ad072a5d9e32b8c7f6dbd0a4d19b4f0b384ce6f58a656bbfa29ff7" Nov 03 02:46:17.811121 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:17.811078 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d8aaa5019ad072a5d9e32b8c7f6dbd0a4d19b4f0b384ce6f58a656bbfa29ff7"} err="failed to get container status \"8d8aaa5019ad072a5d9e32b8c7f6dbd0a4d19b4f0b384ce6f58a656bbfa29ff7\": rpc error: code = NotFound desc = could not find container \"8d8aaa5019ad072a5d9e32b8c7f6dbd0a4d19b4f0b384ce6f58a656bbfa29ff7\": container with ID starting with 8d8aaa5019ad072a5d9e32b8c7f6dbd0a4d19b4f0b384ce6f58a656bbfa29ff7 not found: ID does not exist" Nov 03 02:46:17.968768 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:17.968671 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-b6fb9b-666f59b59f-7d5tv"] Nov 03 02:46:17.974855 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:17.974824 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-b6fb9b-666f59b59f-7d5tv"] Nov 03 02:46:18.090142 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:18.090106 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00f0ee4a-6616-454f-8de4-b39189d45442" path="/var/lib/kubelet/pods/00f0ee4a-6616-454f-8de4-b39189d45442/volumes" Nov 03 02:46:18.090416 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:18.090399 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-b6fb9b-8b66f7f77-5xglx"] Nov 03 02:46:18.092759 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:18.092735 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-b6fb9b-8b66f7f77-5xglx"] Nov 03 02:46:20.087028 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:20.086997 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c2adf4b-1195-4937-b7d1-ea164abf40ab" path="/var/lib/kubelet/pods/0c2adf4b-1195-4937-b7d1-ea164abf40ab/volumes" Nov 03 02:46:23.221354 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:23.221317 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:46:26.799691 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:26.799654 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-b6fb9b-7968d7844-zkbmz"] Nov 03 02:46:26.800208 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:26.799936 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b6fb9b-7968d7844-zkbmz" podUID="444cb387-71d4-4d99-9e81-603e213995cd" containerName="maas-api" containerID="cri-o://61c18fe1222b3838cdd6704dfc0a2ad2bd6380466d10fb2ec963907add2a0910" gracePeriod=30 Nov 03 02:46:26.968188 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:26.968144 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:46:27.055960 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:27.055897 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b6fb9b-7968d7844-zkbmz" Nov 03 02:46:27.178584 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:27.178548 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/444cb387-71d4-4d99-9e81-603e213995cd-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/444cb387-71d4-4d99-9e81-603e213995cd-openshift-service-ca\") pod \"444cb387-71d4-4d99-9e81-603e213995cd\" (UID: \"444cb387-71d4-4d99-9e81-603e213995cd\") " Nov 03 02:46:27.178777 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:27.178600 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/444cb387-71d4-4d99-9e81-603e213995cd-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/444cb387-71d4-4d99-9e81-603e213995cd-maas-api-tls\") pod \"444cb387-71d4-4d99-9e81-603e213995cd\" (UID: \"444cb387-71d4-4d99-9e81-603e213995cd\") " Nov 03 02:46:27.178777 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:27.178657 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/444cb387-71d4-4d99-9e81-603e213995cd-metrics-certs\" (UniqueName: \"kubernetes.io/secret/444cb387-71d4-4d99-9e81-603e213995cd-metrics-certs\") pod \"444cb387-71d4-4d99-9e81-603e213995cd\" (UID: \"444cb387-71d4-4d99-9e81-603e213995cd\") " Nov 03 02:46:27.178908 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:27.178793 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/444cb387-71d4-4d99-9e81-603e213995cd-kube-api-access-d5lf9\" (UniqueName: \"kubernetes.io/projected/444cb387-71d4-4d99-9e81-603e213995cd-kube-api-access-d5lf9\") pod \"444cb387-71d4-4d99-9e81-603e213995cd\" (UID: \"444cb387-71d4-4d99-9e81-603e213995cd\") " Nov 03 02:46:27.179044 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:27.179013 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/444cb387-71d4-4d99-9e81-603e213995cd-openshift-service-ca" pod "444cb387-71d4-4d99-9e81-603e213995cd" (UID: "444cb387-71d4-4d99-9e81-603e213995cd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:46:27.179146 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:27.179128 5437 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/444cb387-71d4-4d99-9e81-603e213995cd-openshift-service-ca\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:46:27.181597 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:27.181565 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/444cb387-71d4-4d99-9e81-603e213995cd-maas-api-tls" pod "444cb387-71d4-4d99-9e81-603e213995cd" (UID: "444cb387-71d4-4d99-9e81-603e213995cd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:46:27.181710 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:27.181631 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/444cb387-71d4-4d99-9e81-603e213995cd-metrics-certs" pod "444cb387-71d4-4d99-9e81-603e213995cd" (UID: "444cb387-71d4-4d99-9e81-603e213995cd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:46:27.181760 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:27.181739 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/444cb387-71d4-4d99-9e81-603e213995cd-kube-api-access-d5lf9" pod "444cb387-71d4-4d99-9e81-603e213995cd" (UID: "444cb387-71d4-4d99-9e81-603e213995cd"). InnerVolumeSpecName "kube-api-access-d5lf9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:46:27.279846 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:27.279813 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5lf9\" (UniqueName: \"kubernetes.io/projected/444cb387-71d4-4d99-9e81-603e213995cd-kube-api-access-d5lf9\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:46:27.279846 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:27.279842 5437 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/444cb387-71d4-4d99-9e81-603e213995cd-maas-api-tls\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:46:27.279846 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:27.279851 5437 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/444cb387-71d4-4d99-9e81-603e213995cd-metrics-certs\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:46:27.845119 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:27.845085 5437 generic.go:356] "Generic (PLEG): container finished" podID="444cb387-71d4-4d99-9e81-603e213995cd" containerID="61c18fe1222b3838cdd6704dfc0a2ad2bd6380466d10fb2ec963907add2a0910" exitCode=0 Nov 03 02:46:27.845611 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:27.845137 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b6fb9b-7968d7844-zkbmz" Nov 03 02:46:27.845611 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:27.845154 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b6fb9b-7968d7844-zkbmz" event={"ID":"444cb387-71d4-4d99-9e81-603e213995cd","Type":"ContainerDied","Data":"61c18fe1222b3838cdd6704dfc0a2ad2bd6380466d10fb2ec963907add2a0910"} Nov 03 02:46:27.845611 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:27.845195 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b6fb9b-7968d7844-zkbmz" event={"ID":"444cb387-71d4-4d99-9e81-603e213995cd","Type":"ContainerDied","Data":"e7dea172b7956882c5a79a9fc01d9ae511fd5bfabdbe8a14931383b212bff40d"} Nov 03 02:46:27.845611 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:27.845217 5437 scope.go:122] "RemoveContainer" containerID="61c18fe1222b3838cdd6704dfc0a2ad2bd6380466d10fb2ec963907add2a0910" Nov 03 02:46:27.854531 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:27.854508 5437 scope.go:122] "RemoveContainer" containerID="61c18fe1222b3838cdd6704dfc0a2ad2bd6380466d10fb2ec963907add2a0910" Nov 03 02:46:27.854778 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:46:27.854759 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61c18fe1222b3838cdd6704dfc0a2ad2bd6380466d10fb2ec963907add2a0910\": container with ID starting with 61c18fe1222b3838cdd6704dfc0a2ad2bd6380466d10fb2ec963907add2a0910 not found: ID does not exist" containerID="61c18fe1222b3838cdd6704dfc0a2ad2bd6380466d10fb2ec963907add2a0910" Nov 03 02:46:27.854833 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:27.854788 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61c18fe1222b3838cdd6704dfc0a2ad2bd6380466d10fb2ec963907add2a0910"} err="failed to get container status \"61c18fe1222b3838cdd6704dfc0a2ad2bd6380466d10fb2ec963907add2a0910\": rpc error: code = NotFound desc = could not find container \"61c18fe1222b3838cdd6704dfc0a2ad2bd6380466d10fb2ec963907add2a0910\": container with ID starting with 61c18fe1222b3838cdd6704dfc0a2ad2bd6380466d10fb2ec963907add2a0910 not found: ID does not exist" Nov 03 02:46:28.014883 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:28.014844 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-b6fb9b-7968d7844-zkbmz"] Nov 03 02:46:28.066582 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:28.066548 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-b6fb9b-7968d7844-zkbmz"] Nov 03 02:46:28.087600 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:28.087571 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="444cb387-71d4-4d99-9e81-603e213995cd" path="/var/lib/kubelet/pods/444cb387-71d4-4d99-9e81-603e213995cd/volumes" Nov 03 02:46:28.916485 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:28.916448 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:46:29.119716 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:29.119673 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:46:34.313095 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:34.313056 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-mxhps"] Nov 03 02:46:34.313527 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:34.313474 5437 cpu_manager.go:414] "containerMap: removing container" podUID="73a40f1f-042c-49ef-ab47-f62e7a1562e0" containerName="cleanup" Nov 03 02:46:34.313527 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:34.313489 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73a40f1f-042c-49ef-ab47-f62e7a1562e0" containerName="cleanup" Nov 03 02:46:34.313527 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:34.313498 5437 cpu_manager.go:414] "containerMap: removing container" podUID="2cac18c4-e538-4c4b-85cf-de232f64de18" containerName="main" Nov 03 02:46:34.313527 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:34.313504 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cac18c4-e538-4c4b-85cf-de232f64de18" containerName="main" Nov 03 02:46:34.313527 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:34.313519 5437 cpu_manager.go:414] "containerMap: removing container" podUID="0c2adf4b-1195-4937-b7d1-ea164abf40ab" containerName="payload-processing" Nov 03 02:46:34.313527 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:34.313524 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c2adf4b-1195-4937-b7d1-ea164abf40ab" containerName="payload-processing" Nov 03 02:46:34.313776 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:34.313534 5437 cpu_manager.go:414] "containerMap: removing container" podUID="73a40f1f-042c-49ef-ab47-f62e7a1562e0" containerName="cleanup" Nov 03 02:46:34.313776 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:34.313538 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73a40f1f-042c-49ef-ab47-f62e7a1562e0" containerName="cleanup" Nov 03 02:46:34.313776 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:34.313546 5437 cpu_manager.go:414] "containerMap: removing container" podUID="444cb387-71d4-4d99-9e81-603e213995cd" containerName="maas-api" Nov 03 02:46:34.313776 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:34.313550 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="444cb387-71d4-4d99-9e81-603e213995cd" containerName="maas-api" Nov 03 02:46:34.313776 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:34.313569 5437 cpu_manager.go:414] "containerMap: removing container" podUID="f4467722-3cdf-4e97-b559-2e5e5b0b62db" containerName="main" Nov 03 02:46:34.313776 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:34.313573 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4467722-3cdf-4e97-b559-2e5e5b0b62db" containerName="main" Nov 03 02:46:34.313776 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:34.313582 5437 cpu_manager.go:414] "containerMap: removing container" podUID="00f0ee4a-6616-454f-8de4-b39189d45442" containerName="payload-pre-processing" Nov 03 02:46:34.313776 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:34.313586 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00f0ee4a-6616-454f-8de4-b39189d45442" containerName="payload-pre-processing" Nov 03 02:46:34.313776 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:34.313657 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4467722-3cdf-4e97-b559-2e5e5b0b62db" containerName="main" Nov 03 02:46:34.313776 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:34.313665 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="00f0ee4a-6616-454f-8de4-b39189d45442" containerName="payload-pre-processing" Nov 03 02:46:34.313776 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:34.313673 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="73a40f1f-042c-49ef-ab47-f62e7a1562e0" containerName="cleanup" Nov 03 02:46:34.313776 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:34.313678 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cac18c4-e538-4c4b-85cf-de232f64de18" containerName="main" Nov 03 02:46:34.313776 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:34.313684 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c2adf4b-1195-4937-b7d1-ea164abf40ab" containerName="payload-processing" Nov 03 02:46:34.313776 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:34.313690 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="73a40f1f-042c-49ef-ab47-f62e7a1562e0" containerName="cleanup" Nov 03 02:46:34.313776 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:34.313695 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="444cb387-71d4-4d99-9e81-603e213995cd" containerName="maas-api" Nov 03 02:46:34.318226 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:34.318190 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-mxhps" Nov 03 02:46:34.336248 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:34.336210 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ba7a2e6-d87a-48b3-a86d-cb8f63c3ddc8-istiod-ca-cert\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-mxhps\" (UID: \"8ba7a2e6-d87a-48b3-a86d-cb8f63c3ddc8\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-mxhps" Nov 03 02:46:34.336248 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:34.336250 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ba7a2e6-d87a-48b3-a86d-cb8f63c3ddc8-workload-socket\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-mxhps\" (UID: \"8ba7a2e6-d87a-48b3-a86d-cb8f63c3ddc8\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-mxhps" Nov 03 02:46:34.336457 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:34.336309 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ba7a2e6-d87a-48b3-a86d-cb8f63c3ddc8-istio-podinfo\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-mxhps\" (UID: \"8ba7a2e6-d87a-48b3-a86d-cb8f63c3ddc8\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-mxhps" Nov 03 02:46:34.336457 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:34.336327 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzpbq\" (UniqueName: \"kubernetes.io/projected/8ba7a2e6-d87a-48b3-a86d-cb8f63c3ddc8-kube-api-access-qzpbq\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-mxhps\" (UID: \"8ba7a2e6-d87a-48b3-a86d-cb8f63c3ddc8\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-mxhps" Nov 03 02:46:34.336457 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:34.336431 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ba7a2e6-d87a-48b3-a86d-cb8f63c3ddc8-workload-certs\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-mxhps\" (UID: \"8ba7a2e6-d87a-48b3-a86d-cb8f63c3ddc8\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-mxhps" Nov 03 02:46:34.336589 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:34.336465 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ba7a2e6-d87a-48b3-a86d-cb8f63c3ddc8-istio-token\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-mxhps\" (UID: \"8ba7a2e6-d87a-48b3-a86d-cb8f63c3ddc8\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-mxhps" Nov 03 02:46:34.336589 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:34.336493 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ba7a2e6-d87a-48b3-a86d-cb8f63c3ddc8-credential-socket\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-mxhps\" (UID: \"8ba7a2e6-d87a-48b3-a86d-cb8f63c3ddc8\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-mxhps" Nov 03 02:46:34.336589 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:34.336570 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ba7a2e6-d87a-48b3-a86d-cb8f63c3ddc8-istio-data\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-mxhps\" (UID: \"8ba7a2e6-d87a-48b3-a86d-cb8f63c3ddc8\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-mxhps" Nov 03 02:46:34.336689 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:34.336596 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ba7a2e6-d87a-48b3-a86d-cb8f63c3ddc8-istio-envoy\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-mxhps\" (UID: \"8ba7a2e6-d87a-48b3-a86d-cb8f63c3ddc8\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-mxhps" Nov 03 02:46:34.373076 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:34.373047 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-b6fb9b-openshift-default-dockercfg-997hh\"" Nov 03 02:46:34.426947 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:34.426902 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-mxhps"] Nov 03 02:46:34.437591 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:34.437548 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ba7a2e6-d87a-48b3-a86d-cb8f63c3ddc8-istiod-ca-cert\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-mxhps\" (UID: \"8ba7a2e6-d87a-48b3-a86d-cb8f63c3ddc8\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-mxhps" Nov 03 02:46:34.437811 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:34.437607 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ba7a2e6-d87a-48b3-a86d-cb8f63c3ddc8-workload-socket\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-mxhps\" (UID: \"8ba7a2e6-d87a-48b3-a86d-cb8f63c3ddc8\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-mxhps" Nov 03 02:46:34.437811 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:34.437642 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ba7a2e6-d87a-48b3-a86d-cb8f63c3ddc8-istio-podinfo\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-mxhps\" (UID: \"8ba7a2e6-d87a-48b3-a86d-cb8f63c3ddc8\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-mxhps" Nov 03 02:46:34.437811 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:34.437668 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzpbq\" (UniqueName: \"kubernetes.io/projected/8ba7a2e6-d87a-48b3-a86d-cb8f63c3ddc8-kube-api-access-qzpbq\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-mxhps\" (UID: \"8ba7a2e6-d87a-48b3-a86d-cb8f63c3ddc8\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-mxhps" Nov 03 02:46:34.437811 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:34.437728 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ba7a2e6-d87a-48b3-a86d-cb8f63c3ddc8-workload-certs\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-mxhps\" (UID: \"8ba7a2e6-d87a-48b3-a86d-cb8f63c3ddc8\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-mxhps" Nov 03 02:46:34.437811 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:34.437751 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ba7a2e6-d87a-48b3-a86d-cb8f63c3ddc8-istio-token\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-mxhps\" (UID: \"8ba7a2e6-d87a-48b3-a86d-cb8f63c3ddc8\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-mxhps" Nov 03 02:46:34.437811 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:34.437792 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ba7a2e6-d87a-48b3-a86d-cb8f63c3ddc8-credential-socket\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-mxhps\" (UID: \"8ba7a2e6-d87a-48b3-a86d-cb8f63c3ddc8\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-mxhps" Nov 03 02:46:34.438189 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:34.437826 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ba7a2e6-d87a-48b3-a86d-cb8f63c3ddc8-istio-data\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-mxhps\" (UID: \"8ba7a2e6-d87a-48b3-a86d-cb8f63c3ddc8\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-mxhps" Nov 03 02:46:34.438189 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:34.437865 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ba7a2e6-d87a-48b3-a86d-cb8f63c3ddc8-istio-envoy\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-mxhps\" (UID: \"8ba7a2e6-d87a-48b3-a86d-cb8f63c3ddc8\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-mxhps" Nov 03 02:46:34.438308 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:34.438214 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ba7a2e6-d87a-48b3-a86d-cb8f63c3ddc8-istiod-ca-cert\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-mxhps\" (UID: \"8ba7a2e6-d87a-48b3-a86d-cb8f63c3ddc8\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-mxhps" Nov 03 02:46:34.438626 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:34.438582 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ba7a2e6-d87a-48b3-a86d-cb8f63c3ddc8-credential-socket\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-mxhps\" (UID: \"8ba7a2e6-d87a-48b3-a86d-cb8f63c3ddc8\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-mxhps" Nov 03 02:46:34.438770 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:34.438678 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ba7a2e6-d87a-48b3-a86d-cb8f63c3ddc8-workload-certs\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-mxhps\" (UID: \"8ba7a2e6-d87a-48b3-a86d-cb8f63c3ddc8\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-mxhps" Nov 03 02:46:34.438842 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:34.438796 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ba7a2e6-d87a-48b3-a86d-cb8f63c3ddc8-istio-data\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-mxhps\" (UID: \"8ba7a2e6-d87a-48b3-a86d-cb8f63c3ddc8\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-mxhps" Nov 03 02:46:34.439122 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:34.439102 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ba7a2e6-d87a-48b3-a86d-cb8f63c3ddc8-workload-socket\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-mxhps\" (UID: \"8ba7a2e6-d87a-48b3-a86d-cb8f63c3ddc8\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-mxhps" Nov 03 02:46:34.441622 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:34.441593 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ba7a2e6-d87a-48b3-a86d-cb8f63c3ddc8-istio-envoy\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-mxhps\" (UID: \"8ba7a2e6-d87a-48b3-a86d-cb8f63c3ddc8\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-mxhps" Nov 03 02:46:34.443880 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:34.443857 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ba7a2e6-d87a-48b3-a86d-cb8f63c3ddc8-istio-podinfo\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-mxhps\" (UID: \"8ba7a2e6-d87a-48b3-a86d-cb8f63c3ddc8\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-mxhps" Nov 03 02:46:34.500575 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:34.500537 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ba7a2e6-d87a-48b3-a86d-cb8f63c3ddc8-istio-token\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-mxhps\" (UID: \"8ba7a2e6-d87a-48b3-a86d-cb8f63c3ddc8\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-mxhps" Nov 03 02:46:34.500748 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:34.500725 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzpbq\" (UniqueName: \"kubernetes.io/projected/8ba7a2e6-d87a-48b3-a86d-cb8f63c3ddc8-kube-api-access-qzpbq\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-mxhps\" (UID: \"8ba7a2e6-d87a-48b3-a86d-cb8f63c3ddc8\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-mxhps" Nov 03 02:46:34.634337 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:34.634248 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-mxhps" Nov 03 02:46:35.054024 ip-10-0-136-16 kubenswrapper[5437]: W1103 02:46:35.053987 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ba7a2e6_d87a_48b3_a86d_cb8f63c3ddc8.slice/crio-2e23cdc8f25fe2e53dfbe55142f8899c859ad8e2a54360e32f08fc7fa800ca3d WatchSource:0}: Error finding container 2e23cdc8f25fe2e53dfbe55142f8899c859ad8e2a54360e32f08fc7fa800ca3d: Status 404 returned error can't find the container with id 2e23cdc8f25fe2e53dfbe55142f8899c859ad8e2a54360e32f08fc7fa800ca3d Nov 03 02:46:35.055175 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:35.055149 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-mxhps"] Nov 03 02:46:35.056466 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:35.056431 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 02:46:35.056560 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:35.056500 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 02:46:35.056560 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:35.056556 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 02:46:35.882649 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:35.882612 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-mxhps" event={"ID":"8ba7a2e6-d87a-48b3-a86d-cb8f63c3ddc8","Type":"ContainerStarted","Data":"ade70e0e45c4cdbcb56129bd0f3f998478af8e9ebfc810f107d3de18a91e0dc7"} Nov 03 02:46:35.883038 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:35.882664 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-mxhps" event={"ID":"8ba7a2e6-d87a-48b3-a86d-cb8f63c3ddc8","Type":"ContainerStarted","Data":"2e23cdc8f25fe2e53dfbe55142f8899c859ad8e2a54360e32f08fc7fa800ca3d"} Nov 03 02:46:35.905095 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:35.905057 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:46:36.112123 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:36.112070 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-mxhps" podStartSLOduration=2.112056639 podStartE2EDuration="2.112056639s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:46:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:46:36.059859484 +0000 UTC m=+5278.089341780" watchObservedRunningTime="2026-11-03 02:46:36.112056639 +0000 UTC m=+5278.141538934" Nov 03 02:46:36.158508 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:36.158428 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:46:36.637388 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:36.636349 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-mxhps" Nov 03 02:46:36.637388 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:36.636535 5437 patch_prober.go:29] interesting pod/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-mxhps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body= Nov 03 02:46:36.637388 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:36.636586 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-mxhps" podUID="8ba7a2e6-d87a-48b3-a86d-cb8f63c3ddc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" Nov 03 02:46:37.637904 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:37.637867 5437 patch_prober.go:29] interesting pod/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-mxhps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body= Nov 03 02:46:37.638526 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:37.638494 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-mxhps" podUID="8ba7a2e6-d87a-48b3-a86d-cb8f63c3ddc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" Nov 03 02:46:38.635349 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:38.635298 5437 patch_prober.go:29] interesting pod/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-mxhps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body= Nov 03 02:46:38.636334 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:38.635724 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-mxhps" podUID="8ba7a2e6-d87a-48b3-a86d-cb8f63c3ddc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" Nov 03 02:46:39.642831 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:39.642779 5437 patch_prober.go:29] interesting pod/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-mxhps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body= Nov 03 02:46:39.643267 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:39.642897 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-mxhps" podUID="8ba7a2e6-d87a-48b3-a86d-cb8f63c3ddc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" Nov 03 02:46:40.178405 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:40.178341 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:46:40.379316 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:40.379283 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:46:40.637780 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:40.637735 5437 patch_prober.go:29] interesting pod/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-mxhps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body= Nov 03 02:46:40.638066 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:40.638034 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-mxhps" podUID="8ba7a2e6-d87a-48b3-a86d-cb8f63c3ddc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" Nov 03 02:46:40.879180 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:40.879145 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:46:41.638749 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:41.638716 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-mxhps" Nov 03 02:46:41.639112 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:41.639085 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-mxhps" Nov 03 02:46:41.639973 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:41.639955 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-mxhps" Nov 03 02:46:43.526060 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:43.526017 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:46:43.980468 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:43.980420 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-mxhps"] Nov 03 02:46:43.980765 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:43.980733 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-mxhps" podUID="8ba7a2e6-d87a-48b3-a86d-cb8f63c3ddc8" containerName="istio-proxy" containerID="cri-o://ade70e0e45c4cdbcb56129bd0f3f998478af8e9ebfc810f107d3de18a91e0dc7" gracePeriod=30 Nov 03 02:46:44.034118 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:46:44.034078 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 02:46:44.042595 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:44.042564 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:46:46.094605 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:46.094568 5437 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-zlql6_1447e14e-f0a8-43bd-8e05-9d32b57fcf34/manager/0.log" Nov 03 02:46:46.547141 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:46.547104 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:46:46.752829 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:46.752788 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:46:49.227549 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:49.227517 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-mxhps" Nov 03 02:46:49.273719 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:49.273635 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ba7a2e6-d87a-48b3-a86d-cb8f63c3ddc8-istio-token\" (UniqueName: \"kubernetes.io/projected/8ba7a2e6-d87a-48b3-a86d-cb8f63c3ddc8-istio-token\") pod \"8ba7a2e6-d87a-48b3-a86d-cb8f63c3ddc8\" (UID: \"8ba7a2e6-d87a-48b3-a86d-cb8f63c3ddc8\") " Nov 03 02:46:49.273719 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:49.273687 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ba7a2e6-d87a-48b3-a86d-cb8f63c3ddc8-kube-api-access-qzpbq\" (UniqueName: \"kubernetes.io/projected/8ba7a2e6-d87a-48b3-a86d-cb8f63c3ddc8-kube-api-access-qzpbq\") pod \"8ba7a2e6-d87a-48b3-a86d-cb8f63c3ddc8\" (UID: \"8ba7a2e6-d87a-48b3-a86d-cb8f63c3ddc8\") " Nov 03 02:46:49.273948 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:49.273744 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ba7a2e6-d87a-48b3-a86d-cb8f63c3ddc8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ba7a2e6-d87a-48b3-a86d-cb8f63c3ddc8-istiod-ca-cert\") pod \"8ba7a2e6-d87a-48b3-a86d-cb8f63c3ddc8\" (UID: \"8ba7a2e6-d87a-48b3-a86d-cb8f63c3ddc8\") " Nov 03 02:46:49.273948 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:49.273788 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ba7a2e6-d87a-48b3-a86d-cb8f63c3ddc8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ba7a2e6-d87a-48b3-a86d-cb8f63c3ddc8-credential-socket\") pod \"8ba7a2e6-d87a-48b3-a86d-cb8f63c3ddc8\" (UID: \"8ba7a2e6-d87a-48b3-a86d-cb8f63c3ddc8\") " Nov 03 02:46:49.273948 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:49.273829 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ba7a2e6-d87a-48b3-a86d-cb8f63c3ddc8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ba7a2e6-d87a-48b3-a86d-cb8f63c3ddc8-istio-data\") pod \"8ba7a2e6-d87a-48b3-a86d-cb8f63c3ddc8\" (UID: \"8ba7a2e6-d87a-48b3-a86d-cb8f63c3ddc8\") " Nov 03 02:46:49.273948 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:49.273876 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ba7a2e6-d87a-48b3-a86d-cb8f63c3ddc8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ba7a2e6-d87a-48b3-a86d-cb8f63c3ddc8-workload-socket\") pod \"8ba7a2e6-d87a-48b3-a86d-cb8f63c3ddc8\" (UID: \"8ba7a2e6-d87a-48b3-a86d-cb8f63c3ddc8\") " Nov 03 02:46:49.273948 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:49.273922 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ba7a2e6-d87a-48b3-a86d-cb8f63c3ddc8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ba7a2e6-d87a-48b3-a86d-cb8f63c3ddc8-istio-envoy\") pod \"8ba7a2e6-d87a-48b3-a86d-cb8f63c3ddc8\" (UID: \"8ba7a2e6-d87a-48b3-a86d-cb8f63c3ddc8\") " Nov 03 02:46:49.274220 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:49.273962 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8ba7a2e6-d87a-48b3-a86d-cb8f63c3ddc8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ba7a2e6-d87a-48b3-a86d-cb8f63c3ddc8-istio-podinfo\") pod \"8ba7a2e6-d87a-48b3-a86d-cb8f63c3ddc8\" (UID: \"8ba7a2e6-d87a-48b3-a86d-cb8f63c3ddc8\") " Nov 03 02:46:49.274220 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:49.274039 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ba7a2e6-d87a-48b3-a86d-cb8f63c3ddc8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ba7a2e6-d87a-48b3-a86d-cb8f63c3ddc8-workload-certs\") pod \"8ba7a2e6-d87a-48b3-a86d-cb8f63c3ddc8\" (UID: \"8ba7a2e6-d87a-48b3-a86d-cb8f63c3ddc8\") " Nov 03 02:46:49.274220 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:49.274138 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ba7a2e6-d87a-48b3-a86d-cb8f63c3ddc8-credential-socket" pod "8ba7a2e6-d87a-48b3-a86d-cb8f63c3ddc8" (UID: "8ba7a2e6-d87a-48b3-a86d-cb8f63c3ddc8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:46:49.274220 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:49.274185 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ba7a2e6-d87a-48b3-a86d-cb8f63c3ddc8-istio-data" pod "8ba7a2e6-d87a-48b3-a86d-cb8f63c3ddc8" (UID: "8ba7a2e6-d87a-48b3-a86d-cb8f63c3ddc8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:46:49.274485 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:49.274232 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ba7a2e6-d87a-48b3-a86d-cb8f63c3ddc8-istiod-ca-cert" pod "8ba7a2e6-d87a-48b3-a86d-cb8f63c3ddc8" (UID: "8ba7a2e6-d87a-48b3-a86d-cb8f63c3ddc8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:46:49.274485 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:49.274437 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ba7a2e6-d87a-48b3-a86d-cb8f63c3ddc8-workload-certs" pod "8ba7a2e6-d87a-48b3-a86d-cb8f63c3ddc8" (UID: "8ba7a2e6-d87a-48b3-a86d-cb8f63c3ddc8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:46:49.274618 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:49.274548 5437 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ba7a2e6-d87a-48b3-a86d-cb8f63c3ddc8-workload-certs\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:46:49.274618 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:49.274576 5437 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ba7a2e6-d87a-48b3-a86d-cb8f63c3ddc8-istiod-ca-cert\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:46:49.274618 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:49.274591 5437 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ba7a2e6-d87a-48b3-a86d-cb8f63c3ddc8-credential-socket\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:46:49.274618 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:49.274604 5437 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ba7a2e6-d87a-48b3-a86d-cb8f63c3ddc8-istio-data\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:46:49.274618 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:49.274540 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ba7a2e6-d87a-48b3-a86d-cb8f63c3ddc8-workload-socket" pod "8ba7a2e6-d87a-48b3-a86d-cb8f63c3ddc8" (UID: "8ba7a2e6-d87a-48b3-a86d-cb8f63c3ddc8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:46:49.275972 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:49.275950 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ba7a2e6-d87a-48b3-a86d-cb8f63c3ddc8-istio-token" pod "8ba7a2e6-d87a-48b3-a86d-cb8f63c3ddc8" (UID: "8ba7a2e6-d87a-48b3-a86d-cb8f63c3ddc8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:46:49.276098 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:49.276019 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ba7a2e6-d87a-48b3-a86d-cb8f63c3ddc8-kube-api-access-qzpbq" pod "8ba7a2e6-d87a-48b3-a86d-cb8f63c3ddc8" (UID: "8ba7a2e6-d87a-48b3-a86d-cb8f63c3ddc8"). InnerVolumeSpecName "kube-api-access-qzpbq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:46:49.276166 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:49.276104 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ba7a2e6-d87a-48b3-a86d-cb8f63c3ddc8-istio-envoy" pod "8ba7a2e6-d87a-48b3-a86d-cb8f63c3ddc8" (UID: "8ba7a2e6-d87a-48b3-a86d-cb8f63c3ddc8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:46:49.276238 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:49.276222 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8ba7a2e6-d87a-48b3-a86d-cb8f63c3ddc8-istio-podinfo" pod "8ba7a2e6-d87a-48b3-a86d-cb8f63c3ddc8" (UID: "8ba7a2e6-d87a-48b3-a86d-cb8f63c3ddc8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:46:49.315938 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:49.315905 5437 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-7b888454db-5q4k2_4de9a87f-21bb-4b1b-9700-0ceef4bbda2c/manager/0.log" Nov 03 02:46:49.375475 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:49.375440 5437 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ba7a2e6-d87a-48b3-a86d-cb8f63c3ddc8-istio-token\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:46:49.375475 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:49.375476 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qzpbq\" (UniqueName: \"kubernetes.io/projected/8ba7a2e6-d87a-48b3-a86d-cb8f63c3ddc8-kube-api-access-qzpbq\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:46:49.375475 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:49.375487 5437 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ba7a2e6-d87a-48b3-a86d-cb8f63c3ddc8-workload-socket\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:46:49.375763 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:49.375497 5437 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ba7a2e6-d87a-48b3-a86d-cb8f63c3ddc8-istio-envoy\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:46:49.375763 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:49.375507 5437 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ba7a2e6-d87a-48b3-a86d-cb8f63c3ddc8-istio-podinfo\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:46:49.956262 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:49.956218 5437 generic.go:356] "Generic (PLEG): container finished" podID="8ba7a2e6-d87a-48b3-a86d-cb8f63c3ddc8" containerID="ade70e0e45c4cdbcb56129bd0f3f998478af8e9ebfc810f107d3de18a91e0dc7" exitCode=0 Nov 03 02:46:49.956262 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:49.956260 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-mxhps" Nov 03 02:46:49.956535 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:49.956305 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-mxhps" event={"ID":"8ba7a2e6-d87a-48b3-a86d-cb8f63c3ddc8","Type":"ContainerDied","Data":"ade70e0e45c4cdbcb56129bd0f3f998478af8e9ebfc810f107d3de18a91e0dc7"} Nov 03 02:46:49.956535 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:49.956385 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-mxhps" event={"ID":"8ba7a2e6-d87a-48b3-a86d-cb8f63c3ddc8","Type":"ContainerDied","Data":"2e23cdc8f25fe2e53dfbe55142f8899c859ad8e2a54360e32f08fc7fa800ca3d"} Nov 03 02:46:49.956535 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:49.956402 5437 scope.go:122] "RemoveContainer" containerID="ade70e0e45c4cdbcb56129bd0f3f998478af8e9ebfc810f107d3de18a91e0dc7" Nov 03 02:46:49.966158 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:49.966141 5437 scope.go:122] "RemoveContainer" containerID="ade70e0e45c4cdbcb56129bd0f3f998478af8e9ebfc810f107d3de18a91e0dc7" Nov 03 02:46:49.966457 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:46:49.966429 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ade70e0e45c4cdbcb56129bd0f3f998478af8e9ebfc810f107d3de18a91e0dc7\": container with ID starting with ade70e0e45c4cdbcb56129bd0f3f998478af8e9ebfc810f107d3de18a91e0dc7 not found: ID does not exist" containerID="ade70e0e45c4cdbcb56129bd0f3f998478af8e9ebfc810f107d3de18a91e0dc7" Nov 03 02:46:49.966563 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:49.966459 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ade70e0e45c4cdbcb56129bd0f3f998478af8e9ebfc810f107d3de18a91e0dc7"} err="failed to get container status \"ade70e0e45c4cdbcb56129bd0f3f998478af8e9ebfc810f107d3de18a91e0dc7\": rpc error: code = NotFound desc = could not find container \"ade70e0e45c4cdbcb56129bd0f3f998478af8e9ebfc810f107d3de18a91e0dc7\": container with ID starting with ade70e0e45c4cdbcb56129bd0f3f998478af8e9ebfc810f107d3de18a91e0dc7 not found: ID does not exist" Nov 03 02:46:49.983769 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:49.983740 5437 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-94pkh_a2288409-ad83-494d-8e5f-17cc3690265e/manager/0.log" Nov 03 02:46:50.126638 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:50.126603 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-mxhps"] Nov 03 02:46:50.178215 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:50.178180 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-mxhps"] Nov 03 02:46:51.234504 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:51.234469 5437 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-f48966749-d28rl_cf2bb627-ab74-4310-8fe9-a499472795f8/manager/1.log" Nov 03 02:46:52.087427 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:52.087383 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ba7a2e6-d87a-48b3-a86d-cb8f63c3ddc8" path="/var/lib/kubelet/pods/8ba7a2e6-d87a-48b3-a86d-cb8f63c3ddc8/volumes" Nov 03 02:46:53.076503 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:53.076464 5437 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-f48966749-d28rl_cf2bb627-ab74-4310-8fe9-a499472795f8/manager/0.log" Nov 03 02:46:57.724325 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:57.724287 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:46:58.344641 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:58.344605 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:46:58.547714 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:46:58.547672 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:47:02.110250 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:47:02.110214 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:47:02.319535 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:47:02.319497 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:47:05.246207 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:47:05.246155 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:47:05.450854 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:47:05.450813 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:47:06.000980 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:47:06.000928 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:47:08.429699 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:47:08.429661 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:47:08.628350 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:47:08.628299 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:47:09.010608 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:47:09.010569 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:47:10.963730 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:47:10.963696 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:47:13.389324 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:47:13.389267 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:47:14.034031 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:47:14.033997 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 02:47:16.804563 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:47:16.804508 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:47:17.056454 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:47:17.056297 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:47:18.279866 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:47:18.279832 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:47:23.068830 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:47:23.068786 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:47:28.118450 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:47:28.118414 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:47:33.092062 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:47:33.092025 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:47:37.977351 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:47:37.977311 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:47:42.707038 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:47:42.706974 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:47:44.034951 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:47:44.034914 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 02:47:47.579285 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:47:47.579242 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:47:50.514580 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:47:50.514549 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:47:50.805471 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:47:50.805424 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:47:52.377029 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:47:52.376994 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:47:56.679830 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:47:56.679791 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:47:58.154961 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:47:58.154917 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:47:58.434050 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:47:58.433950 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:48:00.018727 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:48:00.018686 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:48:00.811247 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:48:00.811202 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:48:04.523667 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:48:04.523630 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-f48966749-d28rl"] Nov 03 02:48:04.524188 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:48:04.523871 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-f48966749-d28rl" podUID="cf2bb627-ab74-4310-8fe9-a499472795f8" containerName="manager" containerID="cri-o://1de541d4a3f5d0328ddcbb85a609e23d4abff77a6803a217c95378692dbc393d" gracePeriod=30 Nov 03 02:48:04.773478 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:48:04.773452 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-f48966749-d28rl" Nov 03 02:48:04.787580 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:48:04.787524 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf2bb627-ab74-4310-8fe9-a499472795f8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf2bb627-ab74-4310-8fe9-a499472795f8-metrics-certs\") pod \"cf2bb627-ab74-4310-8fe9-a499472795f8\" (UID: \"cf2bb627-ab74-4310-8fe9-a499472795f8\") " Nov 03 02:48:04.787678 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:48:04.787650 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf2bb627-ab74-4310-8fe9-a499472795f8-kube-api-access-5xl2g\" (UniqueName: \"kubernetes.io/projected/cf2bb627-ab74-4310-8fe9-a499472795f8-kube-api-access-5xl2g\") pod \"cf2bb627-ab74-4310-8fe9-a499472795f8\" (UID: \"cf2bb627-ab74-4310-8fe9-a499472795f8\") " Nov 03 02:48:04.787741 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:48:04.787700 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf2bb627-ab74-4310-8fe9-a499472795f8-webhook-cert\" (UniqueName: \"kubernetes.io/secret/cf2bb627-ab74-4310-8fe9-a499472795f8-webhook-cert\") pod \"cf2bb627-ab74-4310-8fe9-a499472795f8\" (UID: \"cf2bb627-ab74-4310-8fe9-a499472795f8\") " Nov 03 02:48:04.789655 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:48:04.789625 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf2bb627-ab74-4310-8fe9-a499472795f8-metrics-certs" pod "cf2bb627-ab74-4310-8fe9-a499472795f8" (UID: "cf2bb627-ab74-4310-8fe9-a499472795f8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:48:04.789767 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:48:04.789706 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf2bb627-ab74-4310-8fe9-a499472795f8-webhook-cert" pod "cf2bb627-ab74-4310-8fe9-a499472795f8" (UID: "cf2bb627-ab74-4310-8fe9-a499472795f8"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:48:04.789813 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:48:04.789784 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf2bb627-ab74-4310-8fe9-a499472795f8-kube-api-access-5xl2g" pod "cf2bb627-ab74-4310-8fe9-a499472795f8" (UID: "cf2bb627-ab74-4310-8fe9-a499472795f8"). InnerVolumeSpecName "kube-api-access-5xl2g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:48:04.888730 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:48:04.888690 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xl2g\" (UniqueName: \"kubernetes.io/projected/cf2bb627-ab74-4310-8fe9-a499472795f8-kube-api-access-5xl2g\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:48:04.888730 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:48:04.888723 5437 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cf2bb627-ab74-4310-8fe9-a499472795f8-webhook-cert\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:48:04.888730 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:48:04.888736 5437 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf2bb627-ab74-4310-8fe9-a499472795f8-metrics-certs\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:48:05.311217 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:48:05.311182 5437 generic.go:356] "Generic (PLEG): container finished" podID="cf2bb627-ab74-4310-8fe9-a499472795f8" containerID="1de541d4a3f5d0328ddcbb85a609e23d4abff77a6803a217c95378692dbc393d" exitCode=0 Nov 03 02:48:05.311436 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:48:05.311242 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-f48966749-d28rl" event={"ID":"cf2bb627-ab74-4310-8fe9-a499472795f8","Type":"ContainerDied","Data":"1de541d4a3f5d0328ddcbb85a609e23d4abff77a6803a217c95378692dbc393d"} Nov 03 02:48:05.311436 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:48:05.311247 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-f48966749-d28rl" Nov 03 02:48:05.311436 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:48:05.311285 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-f48966749-d28rl" event={"ID":"cf2bb627-ab74-4310-8fe9-a499472795f8","Type":"ContainerDied","Data":"de15074280b11f046fa0c8bc9f3a016ec98a2afd4311c07ebe4be3a61eb9bc12"} Nov 03 02:48:05.311436 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:48:05.311306 5437 scope.go:122] "RemoveContainer" containerID="1de541d4a3f5d0328ddcbb85a609e23d4abff77a6803a217c95378692dbc393d" Nov 03 02:48:05.321720 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:48:05.321692 5437 scope.go:122] "RemoveContainer" containerID="bbbd8d0e63e183d2910014d617eda5ae2147704e29526c6cc6623f0bacfbbb7c" Nov 03 02:48:05.329750 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:48:05.329733 5437 scope.go:122] "RemoveContainer" containerID="1de541d4a3f5d0328ddcbb85a609e23d4abff77a6803a217c95378692dbc393d" Nov 03 02:48:05.330006 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:48:05.329984 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1de541d4a3f5d0328ddcbb85a609e23d4abff77a6803a217c95378692dbc393d\": container with ID starting with 1de541d4a3f5d0328ddcbb85a609e23d4abff77a6803a217c95378692dbc393d not found: ID does not exist" containerID="1de541d4a3f5d0328ddcbb85a609e23d4abff77a6803a217c95378692dbc393d" Nov 03 02:48:05.330058 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:48:05.330015 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1de541d4a3f5d0328ddcbb85a609e23d4abff77a6803a217c95378692dbc393d"} err="failed to get container status \"1de541d4a3f5d0328ddcbb85a609e23d4abff77a6803a217c95378692dbc393d\": rpc error: code = NotFound desc = could not find container \"1de541d4a3f5d0328ddcbb85a609e23d4abff77a6803a217c95378692dbc393d\": container with ID starting with 1de541d4a3f5d0328ddcbb85a609e23d4abff77a6803a217c95378692dbc393d not found: ID does not exist" Nov 03 02:48:05.330058 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:48:05.330034 5437 scope.go:122] "RemoveContainer" containerID="bbbd8d0e63e183d2910014d617eda5ae2147704e29526c6cc6623f0bacfbbb7c" Nov 03 02:48:05.330251 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:48:05.330235 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbbd8d0e63e183d2910014d617eda5ae2147704e29526c6cc6623f0bacfbbb7c\": container with ID starting with bbbd8d0e63e183d2910014d617eda5ae2147704e29526c6cc6623f0bacfbbb7c not found: ID does not exist" containerID="bbbd8d0e63e183d2910014d617eda5ae2147704e29526c6cc6623f0bacfbbb7c" Nov 03 02:48:05.330296 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:48:05.330256 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbbd8d0e63e183d2910014d617eda5ae2147704e29526c6cc6623f0bacfbbb7c"} err="failed to get container status \"bbbd8d0e63e183d2910014d617eda5ae2147704e29526c6cc6623f0bacfbbb7c\": rpc error: code = NotFound desc = could not find container \"bbbd8d0e63e183d2910014d617eda5ae2147704e29526c6cc6623f0bacfbbb7c\": container with ID starting with bbbd8d0e63e183d2910014d617eda5ae2147704e29526c6cc6623f0bacfbbb7c not found: ID does not exist" Nov 03 02:48:05.480735 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:48:05.480694 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-f48966749-d28rl"] Nov 03 02:48:05.532357 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:48:05.532318 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-f48966749-d28rl"] Nov 03 02:48:06.088204 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:48:06.088168 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf2bb627-ab74-4310-8fe9-a499472795f8" path="/var/lib/kubelet/pods/cf2bb627-ab74-4310-8fe9-a499472795f8/volumes" Nov 03 02:48:06.350041 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:48:06.349957 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:48:06.382056 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:48:06.382019 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-f48966749-cxx48"] Nov 03 02:48:06.382434 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:48:06.382420 5437 cpu_manager.go:414] "containerMap: removing container" podUID="8ba7a2e6-d87a-48b3-a86d-cb8f63c3ddc8" containerName="istio-proxy" Nov 03 02:48:06.382479 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:48:06.382435 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ba7a2e6-d87a-48b3-a86d-cb8f63c3ddc8" containerName="istio-proxy" Nov 03 02:48:06.382479 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:48:06.382454 5437 cpu_manager.go:414] "containerMap: removing container" podUID="cf2bb627-ab74-4310-8fe9-a499472795f8" containerName="manager" Nov 03 02:48:06.382479 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:48:06.382461 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf2bb627-ab74-4310-8fe9-a499472795f8" containerName="manager" Nov 03 02:48:06.382479 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:48:06.382468 5437 cpu_manager.go:414] "containerMap: removing container" podUID="73a40f1f-042c-49ef-ab47-f62e7a1562e0" containerName="cleanup" Nov 03 02:48:06.382479 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:48:06.382473 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73a40f1f-042c-49ef-ab47-f62e7a1562e0" containerName="cleanup" Nov 03 02:48:06.382647 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:48:06.382544 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ba7a2e6-d87a-48b3-a86d-cb8f63c3ddc8" containerName="istio-proxy" Nov 03 02:48:06.382647 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:48:06.382553 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf2bb627-ab74-4310-8fe9-a499472795f8" containerName="manager" Nov 03 02:48:06.382647 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:48:06.382559 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="73a40f1f-042c-49ef-ab47-f62e7a1562e0" containerName="cleanup" Nov 03 02:48:06.382647 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:48:06.382566 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf2bb627-ab74-4310-8fe9-a499472795f8" containerName="manager" Nov 03 02:48:06.386933 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:48:06.386913 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-f48966749-cxx48" Nov 03 02:48:06.400964 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:48:06.400936 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d6302cc5-7ce7-4cd3-8c3e-b733d1c102a4-webhook-cert\") pod \"maas-controller-f48966749-cxx48\" (UID: \"d6302cc5-7ce7-4cd3-8c3e-b733d1c102a4\") " pod="opendatahub/maas-controller-f48966749-cxx48" Nov 03 02:48:06.401114 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:48:06.400968 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6302cc5-7ce7-4cd3-8c3e-b733d1c102a4-metrics-certs\") pod \"maas-controller-f48966749-cxx48\" (UID: \"d6302cc5-7ce7-4cd3-8c3e-b733d1c102a4\") " pod="opendatahub/maas-controller-f48966749-cxx48" Nov 03 02:48:06.401114 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:48:06.401062 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j72np\" (UniqueName: \"kubernetes.io/projected/d6302cc5-7ce7-4cd3-8c3e-b733d1c102a4-kube-api-access-j72np\") pod \"maas-controller-f48966749-cxx48\" (UID: \"d6302cc5-7ce7-4cd3-8c3e-b733d1c102a4\") " pod="opendatahub/maas-controller-f48966749-cxx48" Nov 03 02:48:06.493823 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:48:06.493791 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 03 02:48:06.494003 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:48:06.493917 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 03 02:48:06.494003 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:48:06.493990 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-mbqp2\"" Nov 03 02:48:06.494092 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:48:06.493990 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 03 02:48:06.496320 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:48:06.496297 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-f48966749-cxx48"] Nov 03 02:48:06.502057 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:48:06.502030 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d6302cc5-7ce7-4cd3-8c3e-b733d1c102a4-webhook-cert\") pod \"maas-controller-f48966749-cxx48\" (UID: \"d6302cc5-7ce7-4cd3-8c3e-b733d1c102a4\") " pod="opendatahub/maas-controller-f48966749-cxx48" Nov 03 02:48:06.502185 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:48:06.502062 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6302cc5-7ce7-4cd3-8c3e-b733d1c102a4-metrics-certs\") pod \"maas-controller-f48966749-cxx48\" (UID: \"d6302cc5-7ce7-4cd3-8c3e-b733d1c102a4\") " pod="opendatahub/maas-controller-f48966749-cxx48" Nov 03 02:48:06.502185 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:48:06.502109 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j72np\" (UniqueName: \"kubernetes.io/projected/d6302cc5-7ce7-4cd3-8c3e-b733d1c102a4-kube-api-access-j72np\") pod \"maas-controller-f48966749-cxx48\" (UID: \"d6302cc5-7ce7-4cd3-8c3e-b733d1c102a4\") " pod="opendatahub/maas-controller-f48966749-cxx48" Nov 03 02:48:06.504956 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:48:06.504936 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d6302cc5-7ce7-4cd3-8c3e-b733d1c102a4-webhook-cert\") pod \"maas-controller-f48966749-cxx48\" (UID: \"d6302cc5-7ce7-4cd3-8c3e-b733d1c102a4\") " pod="opendatahub/maas-controller-f48966749-cxx48" Nov 03 02:48:06.505062 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:48:06.504961 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6302cc5-7ce7-4cd3-8c3e-b733d1c102a4-metrics-certs\") pod \"maas-controller-f48966749-cxx48\" (UID: \"d6302cc5-7ce7-4cd3-8c3e-b733d1c102a4\") " pod="opendatahub/maas-controller-f48966749-cxx48" Nov 03 02:48:06.565609 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:48:06.565568 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j72np\" (UniqueName: \"kubernetes.io/projected/d6302cc5-7ce7-4cd3-8c3e-b733d1c102a4-kube-api-access-j72np\") pod \"maas-controller-f48966749-cxx48\" (UID: \"d6302cc5-7ce7-4cd3-8c3e-b733d1c102a4\") " pod="opendatahub/maas-controller-f48966749-cxx48" Nov 03 02:48:06.703022 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:48:06.702935 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-f48966749-cxx48" Nov 03 02:48:07.083176 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:48:07.083136 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-f48966749-cxx48"] Nov 03 02:48:07.101292 ip-10-0-136-16 kubenswrapper[5437]: W1103 02:48:07.101264 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6302cc5_7ce7_4cd3_8c3e_b733d1c102a4.slice/crio-ba45430e95582192902c1bf2cf998ad534ed98b46b4fb6d7b2d005427cb800e9 WatchSource:0}: Error finding container ba45430e95582192902c1bf2cf998ad534ed98b46b4fb6d7b2d005427cb800e9: Status 404 returned error can't find the container with id ba45430e95582192902c1bf2cf998ad534ed98b46b4fb6d7b2d005427cb800e9 Nov 03 02:48:07.102584 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:48:07.102563 5437 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 02:48:07.322982 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:48:07.322946 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-f48966749-cxx48" event={"ID":"d6302cc5-7ce7-4cd3-8c3e-b733d1c102a4","Type":"ContainerStarted","Data":"ba45430e95582192902c1bf2cf998ad534ed98b46b4fb6d7b2d005427cb800e9"} Nov 03 02:48:09.334446 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:48:09.334402 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-f48966749-cxx48" event={"ID":"d6302cc5-7ce7-4cd3-8c3e-b733d1c102a4","Type":"ContainerStarted","Data":"07cbb063d85a905a614855b5972388f67f3effba314249b993c0938d01cc6523"} Nov 03 02:48:09.334910 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:48:09.334543 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-f48966749-cxx48" Nov 03 02:48:09.461307 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:48:09.461245 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-f48966749-cxx48" podStartSLOduration=2.2366695930000002 podStartE2EDuration="3.461230408s" totalImagesPullingTime="1.224560815s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:48:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:48:09.459253792 +0000 UTC m=+5371.488736089" watchObservedRunningTime="2026-11-03 02:48:09.461230408 +0000 UTC m=+5371.490712703" Nov 03 02:48:11.325005 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:48:11.324959 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:48:14.033230 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:48:14.033193 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 02:48:17.062604 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:48:17.062563 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:48:17.266814 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:48:17.266773 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:48:20.342870 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:48:20.342839 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-f48966749-cxx48" Nov 03 02:48:40.706385 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:48:40.706325 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:48:40.777066 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:48:40.777032 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-f48966749-cxx48"] Nov 03 02:48:40.777263 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:48:40.777219 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-f48966749-cxx48" podUID="d6302cc5-7ce7-4cd3-8c3e-b733d1c102a4" containerName="manager" containerID="cri-o://07cbb063d85a905a614855b5972388f67f3effba314249b993c0938d01cc6523" gracePeriod=30 Nov 03 02:48:41.018095 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:48:41.018066 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-f48966749-cxx48" Nov 03 02:48:41.037505 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:48:41.037472 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d6302cc5-7ce7-4cd3-8c3e-b733d1c102a4-webhook-cert\" (UniqueName: \"kubernetes.io/secret/d6302cc5-7ce7-4cd3-8c3e-b733d1c102a4-webhook-cert\") pod \"d6302cc5-7ce7-4cd3-8c3e-b733d1c102a4\" (UID: \"d6302cc5-7ce7-4cd3-8c3e-b733d1c102a4\") " Nov 03 02:48:41.037690 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:48:41.037571 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d6302cc5-7ce7-4cd3-8c3e-b733d1c102a4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6302cc5-7ce7-4cd3-8c3e-b733d1c102a4-metrics-certs\") pod \"d6302cc5-7ce7-4cd3-8c3e-b733d1c102a4\" (UID: \"d6302cc5-7ce7-4cd3-8c3e-b733d1c102a4\") " Nov 03 02:48:41.037690 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:48:41.037617 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6302cc5-7ce7-4cd3-8c3e-b733d1c102a4-kube-api-access-j72np\" (UniqueName: \"kubernetes.io/projected/d6302cc5-7ce7-4cd3-8c3e-b733d1c102a4-kube-api-access-j72np\") pod \"d6302cc5-7ce7-4cd3-8c3e-b733d1c102a4\" (UID: \"d6302cc5-7ce7-4cd3-8c3e-b733d1c102a4\") " Nov 03 02:48:41.039719 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:48:41.039682 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6302cc5-7ce7-4cd3-8c3e-b733d1c102a4-metrics-certs" pod "d6302cc5-7ce7-4cd3-8c3e-b733d1c102a4" (UID: "d6302cc5-7ce7-4cd3-8c3e-b733d1c102a4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:48:41.039941 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:48:41.039907 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6302cc5-7ce7-4cd3-8c3e-b733d1c102a4-webhook-cert" pod "d6302cc5-7ce7-4cd3-8c3e-b733d1c102a4" (UID: "d6302cc5-7ce7-4cd3-8c3e-b733d1c102a4"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:48:41.040138 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:48:41.040113 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6302cc5-7ce7-4cd3-8c3e-b733d1c102a4-kube-api-access-j72np" pod "d6302cc5-7ce7-4cd3-8c3e-b733d1c102a4" (UID: "d6302cc5-7ce7-4cd3-8c3e-b733d1c102a4"). InnerVolumeSpecName "kube-api-access-j72np". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:48:41.138531 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:48:41.138494 5437 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d6302cc5-7ce7-4cd3-8c3e-b733d1c102a4-webhook-cert\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:48:41.138531 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:48:41.138531 5437 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6302cc5-7ce7-4cd3-8c3e-b733d1c102a4-metrics-certs\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:48:41.138780 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:48:41.138546 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j72np\" (UniqueName: \"kubernetes.io/projected/d6302cc5-7ce7-4cd3-8c3e-b733d1c102a4-kube-api-access-j72np\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:48:41.482298 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:48:41.482259 5437 generic.go:356] "Generic (PLEG): container finished" podID="d6302cc5-7ce7-4cd3-8c3e-b733d1c102a4" containerID="07cbb063d85a905a614855b5972388f67f3effba314249b993c0938d01cc6523" exitCode=0 Nov 03 02:48:41.482521 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:48:41.482308 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-f48966749-cxx48" Nov 03 02:48:41.482521 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:48:41.482329 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-f48966749-cxx48" event={"ID":"d6302cc5-7ce7-4cd3-8c3e-b733d1c102a4","Type":"ContainerDied","Data":"07cbb063d85a905a614855b5972388f67f3effba314249b993c0938d01cc6523"} Nov 03 02:48:41.482521 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:48:41.482389 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-f48966749-cxx48" event={"ID":"d6302cc5-7ce7-4cd3-8c3e-b733d1c102a4","Type":"ContainerDied","Data":"ba45430e95582192902c1bf2cf998ad534ed98b46b4fb6d7b2d005427cb800e9"} Nov 03 02:48:41.482521 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:48:41.482418 5437 scope.go:122] "RemoveContainer" containerID="07cbb063d85a905a614855b5972388f67f3effba314249b993c0938d01cc6523" Nov 03 02:48:41.491606 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:48:41.491585 5437 scope.go:122] "RemoveContainer" containerID="07cbb063d85a905a614855b5972388f67f3effba314249b993c0938d01cc6523" Nov 03 02:48:41.491906 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:48:41.491885 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07cbb063d85a905a614855b5972388f67f3effba314249b993c0938d01cc6523\": container with ID starting with 07cbb063d85a905a614855b5972388f67f3effba314249b993c0938d01cc6523 not found: ID does not exist" containerID="07cbb063d85a905a614855b5972388f67f3effba314249b993c0938d01cc6523" Nov 03 02:48:41.491957 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:48:41.491916 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07cbb063d85a905a614855b5972388f67f3effba314249b993c0938d01cc6523"} err="failed to get container status \"07cbb063d85a905a614855b5972388f67f3effba314249b993c0938d01cc6523\": rpc error: code = NotFound desc = could not find container \"07cbb063d85a905a614855b5972388f67f3effba314249b993c0938d01cc6523\": container with ID starting with 07cbb063d85a905a614855b5972388f67f3effba314249b993c0938d01cc6523 not found: ID does not exist" Nov 03 02:48:41.653376 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:48:41.653321 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-f48966749-cxx48"] Nov 03 02:48:41.657720 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:48:41.657690 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-f48966749-cxx48"] Nov 03 02:48:42.088730 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:48:42.088696 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6302cc5-7ce7-4cd3-8c3e-b733d1c102a4" path="/var/lib/kubelet/pods/d6302cc5-7ce7-4cd3-8c3e-b733d1c102a4/volumes" Nov 03 02:48:42.926740 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:48:42.926705 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:48:43.099005 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:48:43.098970 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-f48966749-2xjtr"] Nov 03 02:48:43.099433 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:48:43.099343 5437 cpu_manager.go:414] "containerMap: removing container" podUID="d6302cc5-7ce7-4cd3-8c3e-b733d1c102a4" containerName="manager" Nov 03 02:48:43.099433 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:48:43.099353 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6302cc5-7ce7-4cd3-8c3e-b733d1c102a4" containerName="manager" Nov 03 02:48:43.099433 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:48:43.099401 5437 cpu_manager.go:414] "containerMap: removing container" podUID="cf2bb627-ab74-4310-8fe9-a499472795f8" containerName="manager" Nov 03 02:48:43.099433 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:48:43.099407 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf2bb627-ab74-4310-8fe9-a499472795f8" containerName="manager" Nov 03 02:48:43.099596 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:48:43.099466 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6302cc5-7ce7-4cd3-8c3e-b733d1c102a4" containerName="manager" Nov 03 02:48:43.103996 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:48:43.103975 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-f48966749-2xjtr" Nov 03 02:48:43.157655 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:48:43.157616 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ff7527e0-1356-406f-87b6-68f2e2f4e25d-webhook-cert\") pod \"maas-controller-f48966749-2xjtr\" (UID: \"ff7527e0-1356-406f-87b6-68f2e2f4e25d\") " pod="opendatahub/maas-controller-f48966749-2xjtr" Nov 03 02:48:43.158102 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:48:43.158053 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stbsx\" (UniqueName: \"kubernetes.io/projected/ff7527e0-1356-406f-87b6-68f2e2f4e25d-kube-api-access-stbsx\") pod \"maas-controller-f48966749-2xjtr\" (UID: \"ff7527e0-1356-406f-87b6-68f2e2f4e25d\") " pod="opendatahub/maas-controller-f48966749-2xjtr" Nov 03 02:48:43.158251 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:48:43.158107 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff7527e0-1356-406f-87b6-68f2e2f4e25d-metrics-certs\") pod \"maas-controller-f48966749-2xjtr\" (UID: \"ff7527e0-1356-406f-87b6-68f2e2f4e25d\") " pod="opendatahub/maas-controller-f48966749-2xjtr" Nov 03 02:48:43.212797 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:48:43.212699 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-mbqp2\"" Nov 03 02:48:43.212979 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:48:43.212803 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 03 02:48:43.212979 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:48:43.212908 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 03 02:48:43.213337 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:48:43.213312 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 03 02:48:43.214394 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:48:43.214346 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-f48966749-2xjtr"] Nov 03 02:48:43.259107 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:48:43.259067 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stbsx\" (UniqueName: \"kubernetes.io/projected/ff7527e0-1356-406f-87b6-68f2e2f4e25d-kube-api-access-stbsx\") pod \"maas-controller-f48966749-2xjtr\" (UID: \"ff7527e0-1356-406f-87b6-68f2e2f4e25d\") " pod="opendatahub/maas-controller-f48966749-2xjtr" Nov 03 02:48:43.259279 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:48:43.259112 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff7527e0-1356-406f-87b6-68f2e2f4e25d-metrics-certs\") pod \"maas-controller-f48966749-2xjtr\" (UID: \"ff7527e0-1356-406f-87b6-68f2e2f4e25d\") " pod="opendatahub/maas-controller-f48966749-2xjtr" Nov 03 02:48:43.259279 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:48:43.259147 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ff7527e0-1356-406f-87b6-68f2e2f4e25d-webhook-cert\") pod \"maas-controller-f48966749-2xjtr\" (UID: \"ff7527e0-1356-406f-87b6-68f2e2f4e25d\") " pod="opendatahub/maas-controller-f48966749-2xjtr" Nov 03 02:48:43.261745 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:48:43.261722 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff7527e0-1356-406f-87b6-68f2e2f4e25d-metrics-certs\") pod \"maas-controller-f48966749-2xjtr\" (UID: \"ff7527e0-1356-406f-87b6-68f2e2f4e25d\") " pod="opendatahub/maas-controller-f48966749-2xjtr" Nov 03 02:48:43.261849 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:48:43.261761 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ff7527e0-1356-406f-87b6-68f2e2f4e25d-webhook-cert\") pod \"maas-controller-f48966749-2xjtr\" (UID: \"ff7527e0-1356-406f-87b6-68f2e2f4e25d\") " pod="opendatahub/maas-controller-f48966749-2xjtr" Nov 03 02:48:43.329432 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:48:43.329400 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stbsx\" (UniqueName: \"kubernetes.io/projected/ff7527e0-1356-406f-87b6-68f2e2f4e25d-kube-api-access-stbsx\") pod \"maas-controller-f48966749-2xjtr\" (UID: \"ff7527e0-1356-406f-87b6-68f2e2f4e25d\") " pod="opendatahub/maas-controller-f48966749-2xjtr" Nov 03 02:48:43.416477 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:48:43.416445 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-f48966749-2xjtr" Nov 03 02:48:43.794600 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:48:43.794505 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-f48966749-2xjtr"] Nov 03 02:48:43.820189 ip-10-0-136-16 kubenswrapper[5437]: W1103 02:48:43.820155 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff7527e0_1356_406f_87b6_68f2e2f4e25d.slice/crio-8e03a7fd99da6d68f4e42c8a38e8582a8c53cc49af6b8207bceb40d685d79754 WatchSource:0}: Error finding container 8e03a7fd99da6d68f4e42c8a38e8582a8c53cc49af6b8207bceb40d685d79754: Status 404 returned error can't find the container with id 8e03a7fd99da6d68f4e42c8a38e8582a8c53cc49af6b8207bceb40d685d79754 Nov 03 02:48:44.034483 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:48:44.034359 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 02:48:44.501681 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:48:44.501636 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-f48966749-2xjtr" event={"ID":"ff7527e0-1356-406f-87b6-68f2e2f4e25d","Type":"ContainerStarted","Data":"8e03a7fd99da6d68f4e42c8a38e8582a8c53cc49af6b8207bceb40d685d79754"} Nov 03 02:48:44.536736 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:48:44.536696 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:48:45.508253 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:48:45.508214 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-f48966749-2xjtr" event={"ID":"ff7527e0-1356-406f-87b6-68f2e2f4e25d","Type":"ContainerStarted","Data":"d3137702edc0142efd0803f9c8a52809497969306194fa49b9aef60b76f2fb39"} Nov 03 02:48:45.508780 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:48:45.508339 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-f48966749-2xjtr" Nov 03 02:48:45.632905 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:48:45.632848 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-f48966749-2xjtr" podStartSLOduration=1.7356835739999998 podStartE2EDuration="2.632833414s" totalImagesPullingTime="897.14984ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:48:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:48:45.631875067 +0000 UTC m=+5407.661357361" watchObservedRunningTime="2026-11-03 02:48:45.632833414 +0000 UTC m=+5407.662315719" Nov 03 02:48:56.515500 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:48:56.515414 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-f48966749-2xjtr" Nov 03 02:49:06.736834 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:49:06.736795 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:49:11.669680 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:49:11.669640 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:49:14.033284 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:49:14.033242 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 02:49:31.674218 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:49:31.674177 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-zj6mx"] Nov 03 02:49:31.677569 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:49:31.677550 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-zj6mx" Nov 03 02:49:31.718827 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:49:31.718790 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd1010b6-bf5b-4633-a266-9b5dcf9b1963-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-zj6mx\" (UID: \"fd1010b6-bf5b-4633-a266-9b5dcf9b1963\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-zj6mx" Nov 03 02:49:31.718827 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:49:31.718828 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd1010b6-bf5b-4633-a266-9b5dcf9b1963-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-zj6mx\" (UID: \"fd1010b6-bf5b-4633-a266-9b5dcf9b1963\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-zj6mx" Nov 03 02:49:31.719039 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:49:31.718846 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfsn8\" (UniqueName: \"kubernetes.io/projected/fd1010b6-bf5b-4633-a266-9b5dcf9b1963-kube-api-access-vfsn8\") pod \"maas-default-gateway-openshift-default-59f957fdd7-zj6mx\" (UID: \"fd1010b6-bf5b-4633-a266-9b5dcf9b1963\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-zj6mx" Nov 03 02:49:31.719039 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:49:31.718990 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd1010b6-bf5b-4633-a266-9b5dcf9b1963-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-zj6mx\" (UID: \"fd1010b6-bf5b-4633-a266-9b5dcf9b1963\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-zj6mx" Nov 03 02:49:31.719039 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:49:31.719026 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd1010b6-bf5b-4633-a266-9b5dcf9b1963-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-zj6mx\" (UID: \"fd1010b6-bf5b-4633-a266-9b5dcf9b1963\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-zj6mx" Nov 03 02:49:31.719191 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:49:31.719054 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd1010b6-bf5b-4633-a266-9b5dcf9b1963-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-zj6mx\" (UID: \"fd1010b6-bf5b-4633-a266-9b5dcf9b1963\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-zj6mx" Nov 03 02:49:31.719191 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:49:31.719157 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd1010b6-bf5b-4633-a266-9b5dcf9b1963-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-zj6mx\" (UID: \"fd1010b6-bf5b-4633-a266-9b5dcf9b1963\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-zj6mx" Nov 03 02:49:31.719191 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:49:31.719183 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd1010b6-bf5b-4633-a266-9b5dcf9b1963-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-zj6mx\" (UID: \"fd1010b6-bf5b-4633-a266-9b5dcf9b1963\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-zj6mx" Nov 03 02:49:31.719319 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:49:31.719216 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd1010b6-bf5b-4633-a266-9b5dcf9b1963-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-zj6mx\" (UID: \"fd1010b6-bf5b-4633-a266-9b5dcf9b1963\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-zj6mx" Nov 03 02:49:31.726096 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:49:31.726069 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dbf96"] Nov 03 02:49:31.729708 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:49:31.729693 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dbf96" Nov 03 02:49:31.777824 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:49:31.777795 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-zj6mx"] Nov 03 02:49:31.819727 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:49:31.819690 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd1010b6-bf5b-4633-a266-9b5dcf9b1963-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-zj6mx\" (UID: \"fd1010b6-bf5b-4633-a266-9b5dcf9b1963\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-zj6mx" Nov 03 02:49:31.819914 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:49:31.819736 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbdf72a4-6fc0-4511-b73e-ceabc4a76c2d-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-dbf96\" (UID: \"fbdf72a4-6fc0-4511-b73e-ceabc4a76c2d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dbf96" Nov 03 02:49:31.819914 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:49:31.819762 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd1010b6-bf5b-4633-a266-9b5dcf9b1963-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-zj6mx\" (UID: \"fd1010b6-bf5b-4633-a266-9b5dcf9b1963\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-zj6mx" Nov 03 02:49:31.819914 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:49:31.819794 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2pjj\" (UniqueName: \"kubernetes.io/projected/fbdf72a4-6fc0-4511-b73e-ceabc4a76c2d-kube-api-access-b2pjj\") pod \"maas-default-gateway-openshift-default-59f957fdd7-dbf96\" (UID: \"fbdf72a4-6fc0-4511-b73e-ceabc4a76c2d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dbf96" Nov 03 02:49:31.820115 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:49:31.820092 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd1010b6-bf5b-4633-a266-9b5dcf9b1963-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-zj6mx\" (UID: \"fd1010b6-bf5b-4633-a266-9b5dcf9b1963\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-zj6mx" Nov 03 02:49:31.820178 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:49:31.820155 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd1010b6-bf5b-4633-a266-9b5dcf9b1963-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-zj6mx\" (UID: \"fd1010b6-bf5b-4633-a266-9b5dcf9b1963\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-zj6mx" Nov 03 02:49:31.820325 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:49:31.820299 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd1010b6-bf5b-4633-a266-9b5dcf9b1963-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-zj6mx\" (UID: \"fd1010b6-bf5b-4633-a266-9b5dcf9b1963\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-zj6mx" Nov 03 02:49:31.820488 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:49:31.820343 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbdf72a4-6fc0-4511-b73e-ceabc4a76c2d-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-dbf96\" (UID: \"fbdf72a4-6fc0-4511-b73e-ceabc4a76c2d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dbf96" Nov 03 02:49:31.820488 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:49:31.820463 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd1010b6-bf5b-4633-a266-9b5dcf9b1963-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-zj6mx\" (UID: \"fd1010b6-bf5b-4633-a266-9b5dcf9b1963\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-zj6mx" Nov 03 02:49:31.820660 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:49:31.820499 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbdf72a4-6fc0-4511-b73e-ceabc4a76c2d-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-dbf96\" (UID: \"fbdf72a4-6fc0-4511-b73e-ceabc4a76c2d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dbf96" Nov 03 02:49:31.820660 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:49:31.820522 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd1010b6-bf5b-4633-a266-9b5dcf9b1963-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-zj6mx\" (UID: \"fd1010b6-bf5b-4633-a266-9b5dcf9b1963\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-zj6mx" Nov 03 02:49:31.820660 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:49:31.820548 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfsn8\" (UniqueName: \"kubernetes.io/projected/fd1010b6-bf5b-4633-a266-9b5dcf9b1963-kube-api-access-vfsn8\") pod \"maas-default-gateway-openshift-default-59f957fdd7-zj6mx\" (UID: \"fd1010b6-bf5b-4633-a266-9b5dcf9b1963\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-zj6mx" Nov 03 02:49:31.820660 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:49:31.820580 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbdf72a4-6fc0-4511-b73e-ceabc4a76c2d-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-dbf96\" (UID: \"fbdf72a4-6fc0-4511-b73e-ceabc4a76c2d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dbf96" Nov 03 02:49:31.820660 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:49:31.820618 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbdf72a4-6fc0-4511-b73e-ceabc4a76c2d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-dbf96\" (UID: \"fbdf72a4-6fc0-4511-b73e-ceabc4a76c2d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dbf96" Nov 03 02:49:31.820984 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:49:31.820717 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbdf72a4-6fc0-4511-b73e-ceabc4a76c2d-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-dbf96\" (UID: \"fbdf72a4-6fc0-4511-b73e-ceabc4a76c2d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dbf96" Nov 03 02:49:31.820984 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:49:31.820756 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd1010b6-bf5b-4633-a266-9b5dcf9b1963-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-zj6mx\" (UID: \"fd1010b6-bf5b-4633-a266-9b5dcf9b1963\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-zj6mx" Nov 03 02:49:31.820984 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:49:31.820796 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbdf72a4-6fc0-4511-b73e-ceabc4a76c2d-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-dbf96\" (UID: \"fbdf72a4-6fc0-4511-b73e-ceabc4a76c2d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dbf96" Nov 03 02:49:31.820984 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:49:31.820823 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd1010b6-bf5b-4633-a266-9b5dcf9b1963-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-zj6mx\" (UID: \"fd1010b6-bf5b-4633-a266-9b5dcf9b1963\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-zj6mx" Nov 03 02:49:31.820984 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:49:31.820848 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbdf72a4-6fc0-4511-b73e-ceabc4a76c2d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-dbf96\" (UID: \"fbdf72a4-6fc0-4511-b73e-ceabc4a76c2d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dbf96" Nov 03 02:49:31.820984 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:49:31.820883 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd1010b6-bf5b-4633-a266-9b5dcf9b1963-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-zj6mx\" (UID: \"fd1010b6-bf5b-4633-a266-9b5dcf9b1963\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-zj6mx" Nov 03 02:49:31.821320 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:49:31.821218 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd1010b6-bf5b-4633-a266-9b5dcf9b1963-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-zj6mx\" (UID: \"fd1010b6-bf5b-4633-a266-9b5dcf9b1963\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-zj6mx" Nov 03 02:49:31.821320 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:49:31.821265 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd1010b6-bf5b-4633-a266-9b5dcf9b1963-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-zj6mx\" (UID: \"fd1010b6-bf5b-4633-a266-9b5dcf9b1963\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-zj6mx" Nov 03 02:49:31.821994 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:49:31.821968 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd1010b6-bf5b-4633-a266-9b5dcf9b1963-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-zj6mx\" (UID: \"fd1010b6-bf5b-4633-a266-9b5dcf9b1963\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-zj6mx" Nov 03 02:49:31.823526 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:49:31.823494 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd1010b6-bf5b-4633-a266-9b5dcf9b1963-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-zj6mx\" (UID: \"fd1010b6-bf5b-4633-a266-9b5dcf9b1963\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-zj6mx" Nov 03 02:49:31.823772 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:49:31.823749 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dbf96"] Nov 03 02:49:31.823859 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:49:31.823842 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd1010b6-bf5b-4633-a266-9b5dcf9b1963-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-zj6mx\" (UID: \"fd1010b6-bf5b-4633-a266-9b5dcf9b1963\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-zj6mx" Nov 03 02:49:31.883895 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:49:31.883856 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd1010b6-bf5b-4633-a266-9b5dcf9b1963-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-zj6mx\" (UID: \"fd1010b6-bf5b-4633-a266-9b5dcf9b1963\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-zj6mx" Nov 03 02:49:31.884048 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:49:31.883921 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfsn8\" (UniqueName: \"kubernetes.io/projected/fd1010b6-bf5b-4633-a266-9b5dcf9b1963-kube-api-access-vfsn8\") pod \"maas-default-gateway-openshift-default-59f957fdd7-zj6mx\" (UID: \"fd1010b6-bf5b-4633-a266-9b5dcf9b1963\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-zj6mx" Nov 03 02:49:31.921464 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:49:31.921433 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbdf72a4-6fc0-4511-b73e-ceabc4a76c2d-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-dbf96\" (UID: \"fbdf72a4-6fc0-4511-b73e-ceabc4a76c2d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dbf96" Nov 03 02:49:31.921640 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:49:31.921471 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2pjj\" (UniqueName: \"kubernetes.io/projected/fbdf72a4-6fc0-4511-b73e-ceabc4a76c2d-kube-api-access-b2pjj\") pod \"maas-default-gateway-openshift-default-59f957fdd7-dbf96\" (UID: \"fbdf72a4-6fc0-4511-b73e-ceabc4a76c2d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dbf96" Nov 03 02:49:31.921640 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:49:31.921501 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbdf72a4-6fc0-4511-b73e-ceabc4a76c2d-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-dbf96\" (UID: \"fbdf72a4-6fc0-4511-b73e-ceabc4a76c2d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dbf96" Nov 03 02:49:31.921640 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:49:31.921541 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbdf72a4-6fc0-4511-b73e-ceabc4a76c2d-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-dbf96\" (UID: \"fbdf72a4-6fc0-4511-b73e-ceabc4a76c2d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dbf96" Nov 03 02:49:31.921640 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:49:31.921570 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbdf72a4-6fc0-4511-b73e-ceabc4a76c2d-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-dbf96\" (UID: \"fbdf72a4-6fc0-4511-b73e-ceabc4a76c2d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dbf96" Nov 03 02:49:31.921640 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:49:31.921600 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbdf72a4-6fc0-4511-b73e-ceabc4a76c2d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-dbf96\" (UID: \"fbdf72a4-6fc0-4511-b73e-ceabc4a76c2d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dbf96" Nov 03 02:49:31.921921 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:49:31.921658 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbdf72a4-6fc0-4511-b73e-ceabc4a76c2d-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-dbf96\" (UID: \"fbdf72a4-6fc0-4511-b73e-ceabc4a76c2d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dbf96" Nov 03 02:49:31.921921 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:49:31.921705 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbdf72a4-6fc0-4511-b73e-ceabc4a76c2d-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-dbf96\" (UID: \"fbdf72a4-6fc0-4511-b73e-ceabc4a76c2d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dbf96" Nov 03 02:49:31.921921 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:49:31.921742 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbdf72a4-6fc0-4511-b73e-ceabc4a76c2d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-dbf96\" (UID: \"fbdf72a4-6fc0-4511-b73e-ceabc4a76c2d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dbf96" Nov 03 02:49:31.922079 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:49:31.921986 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbdf72a4-6fc0-4511-b73e-ceabc4a76c2d-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-dbf96\" (UID: \"fbdf72a4-6fc0-4511-b73e-ceabc4a76c2d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dbf96" Nov 03 02:49:31.922079 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:49:31.922064 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbdf72a4-6fc0-4511-b73e-ceabc4a76c2d-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-dbf96\" (UID: \"fbdf72a4-6fc0-4511-b73e-ceabc4a76c2d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dbf96" Nov 03 02:49:31.922256 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:49:31.922234 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbdf72a4-6fc0-4511-b73e-ceabc4a76c2d-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-dbf96\" (UID: \"fbdf72a4-6fc0-4511-b73e-ceabc4a76c2d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dbf96" Nov 03 02:49:31.922451 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:49:31.922424 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbdf72a4-6fc0-4511-b73e-ceabc4a76c2d-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-dbf96\" (UID: \"fbdf72a4-6fc0-4511-b73e-ceabc4a76c2d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dbf96" Nov 03 02:49:31.922646 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:49:31.922627 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbdf72a4-6fc0-4511-b73e-ceabc4a76c2d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-dbf96\" (UID: \"fbdf72a4-6fc0-4511-b73e-ceabc4a76c2d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dbf96" Nov 03 02:49:31.924280 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:49:31.924225 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbdf72a4-6fc0-4511-b73e-ceabc4a76c2d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-dbf96\" (UID: \"fbdf72a4-6fc0-4511-b73e-ceabc4a76c2d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dbf96" Nov 03 02:49:31.924382 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:49:31.924280 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbdf72a4-6fc0-4511-b73e-ceabc4a76c2d-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-dbf96\" (UID: \"fbdf72a4-6fc0-4511-b73e-ceabc4a76c2d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dbf96" Nov 03 02:49:31.984341 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:49:31.984302 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbdf72a4-6fc0-4511-b73e-ceabc4a76c2d-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-dbf96\" (UID: \"fbdf72a4-6fc0-4511-b73e-ceabc4a76c2d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dbf96" Nov 03 02:49:31.984593 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:49:31.984573 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2pjj\" (UniqueName: \"kubernetes.io/projected/fbdf72a4-6fc0-4511-b73e-ceabc4a76c2d-kube-api-access-b2pjj\") pod \"maas-default-gateway-openshift-default-59f957fdd7-dbf96\" (UID: \"fbdf72a4-6fc0-4511-b73e-ceabc4a76c2d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dbf96" Nov 03 02:49:31.990344 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:49:31.990324 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-zj6mx" Nov 03 02:49:32.042464 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:49:32.042437 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dbf96" Nov 03 02:49:32.405261 ip-10-0-136-16 kubenswrapper[5437]: W1103 02:49:32.405227 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd1010b6_bf5b_4633_a266_9b5dcf9b1963.slice/crio-1474bc44aee5ebb892c2436fe7511a7cfaf7f97faf3da1b06a4ddb016e018446 WatchSource:0}: Error finding container 1474bc44aee5ebb892c2436fe7511a7cfaf7f97faf3da1b06a4ddb016e018446: Status 404 returned error can't find the container with id 1474bc44aee5ebb892c2436fe7511a7cfaf7f97faf3da1b06a4ddb016e018446 Nov 03 02:49:32.407936 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:49:32.407898 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 02:49:32.408056 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:49:32.407969 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 02:49:32.408056 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:49:32.408003 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 02:49:32.408515 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:49:32.408429 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-zj6mx"] Nov 03 02:49:32.457431 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:49:32.457401 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dbf96"] Nov 03 02:49:32.457644 ip-10-0-136-16 kubenswrapper[5437]: W1103 02:49:32.457621 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbdf72a4_6fc0_4511_b73e_ceabc4a76c2d.slice/crio-9ec640380562ac18ed5687dc02cbd009ebec96bfebb9b35c2710666f38effd2b WatchSource:0}: Error finding container 9ec640380562ac18ed5687dc02cbd009ebec96bfebb9b35c2710666f38effd2b: Status 404 returned error can't find the container with id 9ec640380562ac18ed5687dc02cbd009ebec96bfebb9b35c2710666f38effd2b Nov 03 02:49:32.459891 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:49:32.459860 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 02:49:32.459983 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:49:32.459924 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 02:49:32.459983 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:49:32.459953 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 02:49:32.731761 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:49:32.731654 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-zj6mx" event={"ID":"fd1010b6-bf5b-4633-a266-9b5dcf9b1963","Type":"ContainerStarted","Data":"b38b42351892cf7faabcb5deaa7ae642260ef906bb89ab9b6ef80731873e9ebc"} Nov 03 02:49:32.731761 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:49:32.731740 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-zj6mx" event={"ID":"fd1010b6-bf5b-4633-a266-9b5dcf9b1963","Type":"ContainerStarted","Data":"1474bc44aee5ebb892c2436fe7511a7cfaf7f97faf3da1b06a4ddb016e018446"} Nov 03 02:49:32.733560 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:49:32.733534 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dbf96" event={"ID":"fbdf72a4-6fc0-4511-b73e-ceabc4a76c2d","Type":"ContainerStarted","Data":"581ec714debc005892146e45a82f9504e849a5bff320798caf442fad62e14289"} Nov 03 02:49:32.733718 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:49:32.733582 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dbf96" event={"ID":"fbdf72a4-6fc0-4511-b73e-ceabc4a76c2d","Type":"ContainerStarted","Data":"9ec640380562ac18ed5687dc02cbd009ebec96bfebb9b35c2710666f38effd2b"} Nov 03 02:49:32.856302 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:49:32.856242 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-zj6mx" podStartSLOduration=1.856226691 podStartE2EDuration="1.856226691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:49:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:49:32.856019549 +0000 UTC m=+5454.885501841" watchObservedRunningTime="2026-11-03 02:49:32.856226691 +0000 UTC m=+5454.885708988" Nov 03 02:49:32.980540 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:49:32.980464 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dbf96" podStartSLOduration=1.980449189 podStartE2EDuration="1.980449189s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:49:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:49:32.978958471 +0000 UTC m=+5455.008440803" watchObservedRunningTime="2026-11-03 02:49:32.980449189 +0000 UTC m=+5455.009931484" Nov 03 02:49:32.991487 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:49:32.991386 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-zj6mx" Nov 03 02:49:33.046532 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:49:33.046495 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dbf96" Nov 03 02:49:33.046901 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:49:33.046875 5437 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-dbf96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body= Nov 03 02:49:33.047068 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:49:33.047044 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dbf96" podUID="fbdf72a4-6fc0-4511-b73e-ceabc4a76c2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" Nov 03 02:49:33.991920 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:49:33.991890 5437 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-zj6mx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body= Nov 03 02:49:33.992388 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:49:33.992348 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-zj6mx" podUID="fd1010b6-bf5b-4633-a266-9b5dcf9b1963" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" Nov 03 02:49:34.045391 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:49:34.044787 5437 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-dbf96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body= Nov 03 02:49:34.045391 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:49:34.044860 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dbf96" podUID="fbdf72a4-6fc0-4511-b73e-ceabc4a76c2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" Nov 03 02:49:34.991740 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:49:34.991656 5437 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-zj6mx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body= Nov 03 02:49:34.991934 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:49:34.991737 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-zj6mx" podUID="fd1010b6-bf5b-4633-a266-9b5dcf9b1963" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" Nov 03 02:49:35.046862 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:49:35.046815 5437 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-dbf96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body= Nov 03 02:49:35.047221 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:49:35.047182 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dbf96" podUID="fbdf72a4-6fc0-4511-b73e-ceabc4a76c2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" Nov 03 02:49:35.990826 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:49:35.990783 5437 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-zj6mx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body= Nov 03 02:49:35.991170 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:49:35.991134 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-zj6mx" podUID="fd1010b6-bf5b-4633-a266-9b5dcf9b1963" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" Nov 03 02:49:36.044389 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:49:36.043689 5437 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-dbf96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body= Nov 03 02:49:36.044389 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:49:36.043755 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dbf96" podUID="fbdf72a4-6fc0-4511-b73e-ceabc4a76c2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" Nov 03 02:49:36.990752 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:49:36.990713 5437 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-zj6mx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body= Nov 03 02:49:36.990972 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:49:36.990779 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-zj6mx" podUID="fd1010b6-bf5b-4633-a266-9b5dcf9b1963" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" Nov 03 02:49:37.044864 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:49:37.044818 5437 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-dbf96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body= Nov 03 02:49:37.045478 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:49:37.045442 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dbf96" podUID="fbdf72a4-6fc0-4511-b73e-ceabc4a76c2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" Nov 03 02:49:37.990853 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:49:37.990815 5437 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-zj6mx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body= Nov 03 02:49:37.991046 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:49:37.990869 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-zj6mx" podUID="fd1010b6-bf5b-4633-a266-9b5dcf9b1963" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" Nov 03 02:49:38.044471 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:49:38.044432 5437 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-dbf96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body= Nov 03 02:49:38.044742 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:49:38.044715 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dbf96" podUID="fbdf72a4-6fc0-4511-b73e-ceabc4a76c2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" Nov 03 02:49:38.990979 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:49:38.990942 5437 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-zj6mx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body= Nov 03 02:49:38.991416 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:49:38.990999 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-zj6mx" podUID="fd1010b6-bf5b-4633-a266-9b5dcf9b1963" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" Nov 03 02:49:39.043837 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:49:39.043798 5437 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-dbf96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body= Nov 03 02:49:39.043981 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:49:39.043863 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dbf96" podUID="fbdf72a4-6fc0-4511-b73e-ceabc4a76c2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" Nov 03 02:49:39.991199 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:49:39.991158 5437 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-zj6mx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body= Nov 03 02:49:39.991635 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:49:39.991218 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-zj6mx" podUID="fd1010b6-bf5b-4633-a266-9b5dcf9b1963" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" Nov 03 02:49:40.043532 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:49:40.043477 5437 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-dbf96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body= Nov 03 02:49:40.043714 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:49:40.043558 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dbf96" podUID="fbdf72a4-6fc0-4511-b73e-ceabc4a76c2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" Nov 03 02:49:42.032699 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:49:42.032647 5437 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-zj6mx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 02:49:42.033250 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:49:42.032724 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-zj6mx" podUID="fd1010b6-bf5b-4633-a266-9b5dcf9b1963" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 02:49:42.084592 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:49:42.084547 5437 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-dbf96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 02:49:42.084847 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:49:42.084816 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dbf96" podUID="fbdf72a4-6fc0-4511-b73e-ceabc4a76c2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 02:49:43.073660 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:49:43.073610 5437 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-zj6mx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 02:49:43.074207 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:49:43.073685 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-zj6mx" podUID="fd1010b6-bf5b-4633-a266-9b5dcf9b1963" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 02:49:43.127553 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:49:43.127499 5437 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-dbf96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 02:49:43.127765 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:49:43.127581 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dbf96" podUID="fbdf72a4-6fc0-4511-b73e-ceabc4a76c2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 02:49:43.135801 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:49:43.135770 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-zj6mx" Nov 03 02:49:43.136253 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:49:43.136230 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-zj6mx" Nov 03 02:49:43.137179 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:49:43.137156 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-zj6mx" Nov 03 02:49:44.039294 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:49:44.039250 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 02:49:44.128600 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:49:44.128551 5437 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-dbf96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 02:49:44.129028 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:49:44.128648 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dbf96" podUID="fbdf72a4-6fc0-4511-b73e-ceabc4a76c2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 02:49:45.180388 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:49:45.179484 5437 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-dbf96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 02:49:45.180388 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:49:45.179585 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dbf96" podUID="fbdf72a4-6fc0-4511-b73e-ceabc4a76c2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 02:49:46.231389 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:49:46.229514 5437 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-dbf96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 02:49:46.231389 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:49:46.229618 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dbf96" podUID="fbdf72a4-6fc0-4511-b73e-ceabc4a76c2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 02:49:46.479911 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:49:46.479874 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dbf96" Nov 03 02:49:46.480302 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:49:46.480280 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dbf96" Nov 03 02:49:46.480947 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:49:46.480926 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dbf96" Nov 03 02:50:07.130469 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:50:07.130316 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:50:12.047920 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:50:12.047884 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:50:14.034709 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:50:14.034670 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 02:50:34.738766 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:50:34.738723 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:50:37.785118 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:50:37.785070 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-prio-pool-1bfe99e1-kserve-57b57f597c-sfqnk"] Nov 03 02:50:37.789234 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:50:37.789214 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-1bfe99e1-kserve-57b57f597c-sfqnk" Nov 03 02:50:37.795204 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:50:37.795178 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18610cda-0970-4c77-a0aa-feaf2f0743af-tls-certs\") pod \"e2e-prio-pool-1bfe99e1-kserve-57b57f597c-sfqnk\" (UID: \"18610cda-0970-4c77-a0aa-feaf2f0743af\") " pod="llm/e2e-prio-pool-1bfe99e1-kserve-57b57f597c-sfqnk" Nov 03 02:50:37.795347 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:50:37.795246 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18610cda-0970-4c77-a0aa-feaf2f0743af-model-cache\") pod \"e2e-prio-pool-1bfe99e1-kserve-57b57f597c-sfqnk\" (UID: \"18610cda-0970-4c77-a0aa-feaf2f0743af\") " pod="llm/e2e-prio-pool-1bfe99e1-kserve-57b57f597c-sfqnk" Nov 03 02:50:37.795347 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:50:37.795314 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5d5t\" (UniqueName: \"kubernetes.io/projected/18610cda-0970-4c77-a0aa-feaf2f0743af-kube-api-access-b5d5t\") pod \"e2e-prio-pool-1bfe99e1-kserve-57b57f597c-sfqnk\" (UID: \"18610cda-0970-4c77-a0aa-feaf2f0743af\") " pod="llm/e2e-prio-pool-1bfe99e1-kserve-57b57f597c-sfqnk" Nov 03 02:50:37.795498 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:50:37.795349 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18610cda-0970-4c77-a0aa-feaf2f0743af-tmp-dir\") pod \"e2e-prio-pool-1bfe99e1-kserve-57b57f597c-sfqnk\" (UID: \"18610cda-0970-4c77-a0aa-feaf2f0743af\") " pod="llm/e2e-prio-pool-1bfe99e1-kserve-57b57f597c-sfqnk" Nov 03 02:50:37.795498 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:50:37.795388 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18610cda-0970-4c77-a0aa-feaf2f0743af-dshm\") pod \"e2e-prio-pool-1bfe99e1-kserve-57b57f597c-sfqnk\" (UID: \"18610cda-0970-4c77-a0aa-feaf2f0743af\") " pod="llm/e2e-prio-pool-1bfe99e1-kserve-57b57f597c-sfqnk" Nov 03 02:50:37.795498 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:50:37.795449 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18610cda-0970-4c77-a0aa-feaf2f0743af-home\") pod \"e2e-prio-pool-1bfe99e1-kserve-57b57f597c-sfqnk\" (UID: \"18610cda-0970-4c77-a0aa-feaf2f0743af\") " pod="llm/e2e-prio-pool-1bfe99e1-kserve-57b57f597c-sfqnk" Nov 03 02:50:37.896326 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:50:37.896286 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5d5t\" (UniqueName: \"kubernetes.io/projected/18610cda-0970-4c77-a0aa-feaf2f0743af-kube-api-access-b5d5t\") pod \"e2e-prio-pool-1bfe99e1-kserve-57b57f597c-sfqnk\" (UID: \"18610cda-0970-4c77-a0aa-feaf2f0743af\") " pod="llm/e2e-prio-pool-1bfe99e1-kserve-57b57f597c-sfqnk" Nov 03 02:50:37.896537 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:50:37.896332 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18610cda-0970-4c77-a0aa-feaf2f0743af-tmp-dir\") pod \"e2e-prio-pool-1bfe99e1-kserve-57b57f597c-sfqnk\" (UID: \"18610cda-0970-4c77-a0aa-feaf2f0743af\") " pod="llm/e2e-prio-pool-1bfe99e1-kserve-57b57f597c-sfqnk" Nov 03 02:50:37.896537 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:50:37.896356 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18610cda-0970-4c77-a0aa-feaf2f0743af-dshm\") pod \"e2e-prio-pool-1bfe99e1-kserve-57b57f597c-sfqnk\" (UID: \"18610cda-0970-4c77-a0aa-feaf2f0743af\") " pod="llm/e2e-prio-pool-1bfe99e1-kserve-57b57f597c-sfqnk" Nov 03 02:50:37.896537 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:50:37.896405 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18610cda-0970-4c77-a0aa-feaf2f0743af-home\") pod \"e2e-prio-pool-1bfe99e1-kserve-57b57f597c-sfqnk\" (UID: \"18610cda-0970-4c77-a0aa-feaf2f0743af\") " pod="llm/e2e-prio-pool-1bfe99e1-kserve-57b57f597c-sfqnk" Nov 03 02:50:37.896537 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:50:37.896445 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18610cda-0970-4c77-a0aa-feaf2f0743af-tls-certs\") pod \"e2e-prio-pool-1bfe99e1-kserve-57b57f597c-sfqnk\" (UID: \"18610cda-0970-4c77-a0aa-feaf2f0743af\") " pod="llm/e2e-prio-pool-1bfe99e1-kserve-57b57f597c-sfqnk" Nov 03 02:50:37.896537 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:50:37.896522 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18610cda-0970-4c77-a0aa-feaf2f0743af-model-cache\") pod \"e2e-prio-pool-1bfe99e1-kserve-57b57f597c-sfqnk\" (UID: \"18610cda-0970-4c77-a0aa-feaf2f0743af\") " pod="llm/e2e-prio-pool-1bfe99e1-kserve-57b57f597c-sfqnk" Nov 03 02:50:37.896922 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:50:37.896892 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18610cda-0970-4c77-a0aa-feaf2f0743af-tmp-dir\") pod \"e2e-prio-pool-1bfe99e1-kserve-57b57f597c-sfqnk\" (UID: \"18610cda-0970-4c77-a0aa-feaf2f0743af\") " pod="llm/e2e-prio-pool-1bfe99e1-kserve-57b57f597c-sfqnk" Nov 03 02:50:37.897002 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:50:37.896909 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18610cda-0970-4c77-a0aa-feaf2f0743af-home\") pod \"e2e-prio-pool-1bfe99e1-kserve-57b57f597c-sfqnk\" (UID: \"18610cda-0970-4c77-a0aa-feaf2f0743af\") " pod="llm/e2e-prio-pool-1bfe99e1-kserve-57b57f597c-sfqnk" Nov 03 02:50:37.897002 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:50:37.896932 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18610cda-0970-4c77-a0aa-feaf2f0743af-model-cache\") pod \"e2e-prio-pool-1bfe99e1-kserve-57b57f597c-sfqnk\" (UID: \"18610cda-0970-4c77-a0aa-feaf2f0743af\") " pod="llm/e2e-prio-pool-1bfe99e1-kserve-57b57f597c-sfqnk" Nov 03 02:50:37.898783 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:50:37.898759 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18610cda-0970-4c77-a0aa-feaf2f0743af-dshm\") pod \"e2e-prio-pool-1bfe99e1-kserve-57b57f597c-sfqnk\" (UID: \"18610cda-0970-4c77-a0aa-feaf2f0743af\") " pod="llm/e2e-prio-pool-1bfe99e1-kserve-57b57f597c-sfqnk" Nov 03 02:50:37.899630 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:50:37.899610 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-1bfe99e1-kserve-57b57f597c-sfqnk"] Nov 03 02:50:37.910027 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:50:37.910004 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-prio-pool-1bfe99e1-kserve-self-signed-certs\"" Nov 03 02:50:37.920128 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:50:37.920100 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18610cda-0970-4c77-a0aa-feaf2f0743af-tls-certs\") pod \"e2e-prio-pool-1bfe99e1-kserve-57b57f597c-sfqnk\" (UID: \"18610cda-0970-4c77-a0aa-feaf2f0743af\") " pod="llm/e2e-prio-pool-1bfe99e1-kserve-57b57f597c-sfqnk" Nov 03 02:50:37.964878 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:50:37.964719 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:50:37.965574 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:50:37.965549 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5d5t\" (UniqueName: \"kubernetes.io/projected/18610cda-0970-4c77-a0aa-feaf2f0743af-kube-api-access-b5d5t\") pod \"e2e-prio-pool-1bfe99e1-kserve-57b57f597c-sfqnk\" (UID: \"18610cda-0970-4c77-a0aa-feaf2f0743af\") " pod="llm/e2e-prio-pool-1bfe99e1-kserve-57b57f597c-sfqnk" Nov 03 02:50:38.101433 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:50:38.101354 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-1bfe99e1-kserve-57b57f597c-sfqnk" Nov 03 02:50:38.510308 ip-10-0-136-16 kubenswrapper[5437]: W1103 02:50:38.510277 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod18610cda_0970_4c77_a0aa_feaf2f0743af.slice/crio-407b57a428ef2846a260bf405009f2501c2e7852724dbe574bb21cb8d0f9587b WatchSource:0}: Error finding container 407b57a428ef2846a260bf405009f2501c2e7852724dbe574bb21cb8d0f9587b: Status 404 returned error can't find the container with id 407b57a428ef2846a260bf405009f2501c2e7852724dbe574bb21cb8d0f9587b Nov 03 02:50:38.511565 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:50:38.511537 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-1bfe99e1-kserve-57b57f597c-sfqnk"] Nov 03 02:50:38.742521 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:50:38.742459 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:50:39.162325 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:50:39.162286 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-1bfe99e1-kserve-57b57f597c-sfqnk" event={"ID":"18610cda-0970-4c77-a0aa-feaf2f0743af","Type":"ContainerStarted","Data":"317aeea8bc1202136e4699a4c59e0de0fd5a03ab3ad0538ad8a7b8e7c22fbfa0"} Nov 03 02:50:39.162879 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:50:39.162338 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-1bfe99e1-kserve-57b57f597c-sfqnk" event={"ID":"18610cda-0970-4c77-a0aa-feaf2f0743af","Type":"ContainerStarted","Data":"407b57a428ef2846a260bf405009f2501c2e7852724dbe574bb21cb8d0f9587b"} Nov 03 02:50:40.102254 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:50:40.102216 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-prio-pool-1bfe99e1-kserve-57b57f597c-sfqnk" Nov 03 02:50:42.738282 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:50:42.738242 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:50:44.098734 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:50:44.098063 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 02:50:48.102160 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:50:48.102124 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-prio-pool-1bfe99e1-kserve-57b57f597c-sfqnk" Nov 03 02:50:48.114556 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:50:48.114526 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-prio-pool-1bfe99e1-kserve-57b57f597c-sfqnk" Nov 03 02:50:48.228012 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:50:48.227979 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-prio-pool-1bfe99e1-kserve-57b57f597c-sfqnk" Nov 03 02:50:48.237106 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:50:48.237060 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-prio-pool-1bfe99e1-kserve-57b57f597c-sfqnk" podStartSLOduration=11.237047365 podStartE2EDuration="11.237047365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:50:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:50:39.294606504 +0000 UTC m=+5521.324088810" watchObservedRunningTime="2026-11-03 02:50:48.237047365 +0000 UTC m=+5530.266529659" Nov 03 02:51:07.426910 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:51:07.426866 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:51:12.356779 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:51:12.356739 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:51:14.034982 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:51:14.034939 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 02:51:44.035006 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:51:44.034968 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 02:51:46.381705 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:51:46.381672 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-1bfe99e1-kserve-57b57f597c-sfqnk"] Nov 03 02:51:46.461685 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:51:46.461650 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:51:46.775034 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:51:46.774933 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:52:01.403027 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:01.402989 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-prio-pool-1bfe99e1-kserve-57b57f597c-sfqnk" podUID="18610cda-0970-4c77-a0aa-feaf2f0743af" containerName="main" containerID="cri-o://317aeea8bc1202136e4699a4c59e0de0fd5a03ab3ad0538ad8a7b8e7c22fbfa0" gracePeriod=45 Nov 03 02:52:01.609181 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:01.609149 5437 generic.go:356] "Generic (PLEG): container finished" podID="18610cda-0970-4c77-a0aa-feaf2f0743af" containerID="317aeea8bc1202136e4699a4c59e0de0fd5a03ab3ad0538ad8a7b8e7c22fbfa0" exitCode=0 Nov 03 02:52:01.609395 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:01.609227 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-1bfe99e1-kserve-57b57f597c-sfqnk" event={"ID":"18610cda-0970-4c77-a0aa-feaf2f0743af","Type":"ContainerDied","Data":"317aeea8bc1202136e4699a4c59e0de0fd5a03ab3ad0538ad8a7b8e7c22fbfa0"} Nov 03 02:52:01.651386 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:01.651348 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-1bfe99e1-kserve-57b57f597c-sfqnk" Nov 03 02:52:01.781393 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:01.781298 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18610cda-0970-4c77-a0aa-feaf2f0743af-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18610cda-0970-4c77-a0aa-feaf2f0743af-model-cache\") pod \"18610cda-0970-4c77-a0aa-feaf2f0743af\" (UID: \"18610cda-0970-4c77-a0aa-feaf2f0743af\") " Nov 03 02:52:01.781393 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:01.781348 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18610cda-0970-4c77-a0aa-feaf2f0743af-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18610cda-0970-4c77-a0aa-feaf2f0743af-tmp-dir\") pod \"18610cda-0970-4c77-a0aa-feaf2f0743af\" (UID: \"18610cda-0970-4c77-a0aa-feaf2f0743af\") " Nov 03 02:52:01.781624 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:01.781465 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18610cda-0970-4c77-a0aa-feaf2f0743af-home\" (UniqueName: \"kubernetes.io/empty-dir/18610cda-0970-4c77-a0aa-feaf2f0743af-home\") pod \"18610cda-0970-4c77-a0aa-feaf2f0743af\" (UID: \"18610cda-0970-4c77-a0aa-feaf2f0743af\") " Nov 03 02:52:01.781624 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:01.781531 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18610cda-0970-4c77-a0aa-feaf2f0743af-dshm\" (UniqueName: \"kubernetes.io/empty-dir/18610cda-0970-4c77-a0aa-feaf2f0743af-dshm\") pod \"18610cda-0970-4c77-a0aa-feaf2f0743af\" (UID: \"18610cda-0970-4c77-a0aa-feaf2f0743af\") " Nov 03 02:52:01.781624 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:01.781597 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18610cda-0970-4c77-a0aa-feaf2f0743af-kube-api-access-b5d5t\" (UniqueName: \"kubernetes.io/projected/18610cda-0970-4c77-a0aa-feaf2f0743af-kube-api-access-b5d5t\") pod \"18610cda-0970-4c77-a0aa-feaf2f0743af\" (UID: \"18610cda-0970-4c77-a0aa-feaf2f0743af\") " Nov 03 02:52:01.781860 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:01.781625 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18610cda-0970-4c77-a0aa-feaf2f0743af-tls-certs\" (UniqueName: \"kubernetes.io/secret/18610cda-0970-4c77-a0aa-feaf2f0743af-tls-certs\") pod \"18610cda-0970-4c77-a0aa-feaf2f0743af\" (UID: \"18610cda-0970-4c77-a0aa-feaf2f0743af\") " Nov 03 02:52:01.781860 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:01.781652 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18610cda-0970-4c77-a0aa-feaf2f0743af-model-cache" pod "18610cda-0970-4c77-a0aa-feaf2f0743af" (UID: "18610cda-0970-4c77-a0aa-feaf2f0743af"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:52:01.781860 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:01.781711 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18610cda-0970-4c77-a0aa-feaf2f0743af-tmp-dir" pod "18610cda-0970-4c77-a0aa-feaf2f0743af" (UID: "18610cda-0970-4c77-a0aa-feaf2f0743af"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:52:01.781860 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:01.781795 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18610cda-0970-4c77-a0aa-feaf2f0743af-home" pod "18610cda-0970-4c77-a0aa-feaf2f0743af" (UID: "18610cda-0970-4c77-a0aa-feaf2f0743af"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:52:01.782068 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:01.782026 5437 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18610cda-0970-4c77-a0aa-feaf2f0743af-home\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:52:01.782068 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:01.782047 5437 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18610cda-0970-4c77-a0aa-feaf2f0743af-model-cache\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:52:01.782068 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:01.782063 5437 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18610cda-0970-4c77-a0aa-feaf2f0743af-tmp-dir\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:52:01.783914 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:01.783888 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18610cda-0970-4c77-a0aa-feaf2f0743af-kube-api-access-b5d5t" pod "18610cda-0970-4c77-a0aa-feaf2f0743af" (UID: "18610cda-0970-4c77-a0aa-feaf2f0743af"). InnerVolumeSpecName "kube-api-access-b5d5t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:52:01.784175 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:01.784161 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18610cda-0970-4c77-a0aa-feaf2f0743af-dshm" pod "18610cda-0970-4c77-a0aa-feaf2f0743af" (UID: "18610cda-0970-4c77-a0aa-feaf2f0743af"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:52:01.784227 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:01.784202 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18610cda-0970-4c77-a0aa-feaf2f0743af-tls-certs" pod "18610cda-0970-4c77-a0aa-feaf2f0743af" (UID: "18610cda-0970-4c77-a0aa-feaf2f0743af"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:52:01.882810 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:01.882774 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5d5t\" (UniqueName: \"kubernetes.io/projected/18610cda-0970-4c77-a0aa-feaf2f0743af-kube-api-access-b5d5t\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:52:01.882810 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:01.882806 5437 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18610cda-0970-4c77-a0aa-feaf2f0743af-tls-certs\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:52:01.882810 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:01.882815 5437 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18610cda-0970-4c77-a0aa-feaf2f0743af-dshm\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:52:02.615811 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:02.615780 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-1bfe99e1-kserve-57b57f597c-sfqnk" Nov 03 02:52:02.616270 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:02.615776 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-1bfe99e1-kserve-57b57f597c-sfqnk" event={"ID":"18610cda-0970-4c77-a0aa-feaf2f0743af","Type":"ContainerDied","Data":"407b57a428ef2846a260bf405009f2501c2e7852724dbe574bb21cb8d0f9587b"} Nov 03 02:52:02.616270 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:02.615906 5437 scope.go:122] "RemoveContainer" containerID="317aeea8bc1202136e4699a4c59e0de0fd5a03ab3ad0538ad8a7b8e7c22fbfa0" Nov 03 02:52:02.785563 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:02.785525 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-1bfe99e1-kserve-57b57f597c-sfqnk"] Nov 03 02:52:02.837193 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:02.837144 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-prio-pool-1bfe99e1-kserve-57b57f597c-sfqnk"] Nov 03 02:52:04.087783 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:04.087737 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18610cda-0970-4c77-a0aa-feaf2f0743af" path="/var/lib/kubelet/pods/18610cda-0970-4c77-a0aa-feaf2f0743af/volumes" Nov 03 02:52:04.834783 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:04.834743 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:52:05.301129 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:05.301097 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-65a246-openshift-default-7d4797c5b55m9qb"] Nov 03 02:52:05.301563 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:05.301548 5437 cpu_manager.go:414] "containerMap: removing container" podUID="18610cda-0970-4c77-a0aa-feaf2f0743af" containerName="main" Nov 03 02:52:05.301613 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:05.301564 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18610cda-0970-4c77-a0aa-feaf2f0743af" containerName="main" Nov 03 02:52:05.301657 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:05.301623 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="18610cda-0970-4c77-a0aa-feaf2f0743af" containerName="main" Nov 03 02:52:05.306239 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:05.306220 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-65a246-openshift-default-7d4797c5b55m9qb" Nov 03 02:52:05.360916 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:05.360879 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-65a246-openshift-default-dockercfg-22fft\"" Nov 03 02:52:05.414815 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:05.414780 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-65a246-openshift-default-7d4797c5b55m9qb"] Nov 03 02:52:05.417255 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:05.417221 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de4179d8-a136-4ddd-a0b2-063f250ce5ad-istiod-ca-cert\") pod \"e2e-worker-single-main-65a246-openshift-default-7d4797c5b55m9qb\" (UID: \"de4179d8-a136-4ddd-a0b2-063f250ce5ad\") " pod="openshift-ingress/e2e-worker-single-main-65a246-openshift-default-7d4797c5b55m9qb" Nov 03 02:52:05.417428 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:05.417275 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de4179d8-a136-4ddd-a0b2-063f250ce5ad-istio-token\") pod \"e2e-worker-single-main-65a246-openshift-default-7d4797c5b55m9qb\" (UID: \"de4179d8-a136-4ddd-a0b2-063f250ce5ad\") " pod="openshift-ingress/e2e-worker-single-main-65a246-openshift-default-7d4797c5b55m9qb" Nov 03 02:52:05.417428 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:05.417350 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2tp7\" (UniqueName: \"kubernetes.io/projected/de4179d8-a136-4ddd-a0b2-063f250ce5ad-kube-api-access-x2tp7\") pod \"e2e-worker-single-main-65a246-openshift-default-7d4797c5b55m9qb\" (UID: \"de4179d8-a136-4ddd-a0b2-063f250ce5ad\") " pod="openshift-ingress/e2e-worker-single-main-65a246-openshift-default-7d4797c5b55m9qb" Nov 03 02:52:05.417562 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:05.417456 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de4179d8-a136-4ddd-a0b2-063f250ce5ad-istio-envoy\") pod \"e2e-worker-single-main-65a246-openshift-default-7d4797c5b55m9qb\" (UID: \"de4179d8-a136-4ddd-a0b2-063f250ce5ad\") " pod="openshift-ingress/e2e-worker-single-main-65a246-openshift-default-7d4797c5b55m9qb" Nov 03 02:52:05.417562 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:05.417540 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de4179d8-a136-4ddd-a0b2-063f250ce5ad-credential-socket\") pod \"e2e-worker-single-main-65a246-openshift-default-7d4797c5b55m9qb\" (UID: \"de4179d8-a136-4ddd-a0b2-063f250ce5ad\") " pod="openshift-ingress/e2e-worker-single-main-65a246-openshift-default-7d4797c5b55m9qb" Nov 03 02:52:05.417667 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:05.417626 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de4179d8-a136-4ddd-a0b2-063f250ce5ad-istio-data\") pod \"e2e-worker-single-main-65a246-openshift-default-7d4797c5b55m9qb\" (UID: \"de4179d8-a136-4ddd-a0b2-063f250ce5ad\") " pod="openshift-ingress/e2e-worker-single-main-65a246-openshift-default-7d4797c5b55m9qb" Nov 03 02:52:05.417776 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:05.417753 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de4179d8-a136-4ddd-a0b2-063f250ce5ad-workload-certs\") pod \"e2e-worker-single-main-65a246-openshift-default-7d4797c5b55m9qb\" (UID: \"de4179d8-a136-4ddd-a0b2-063f250ce5ad\") " pod="openshift-ingress/e2e-worker-single-main-65a246-openshift-default-7d4797c5b55m9qb" Nov 03 02:52:05.417827 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:05.417808 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de4179d8-a136-4ddd-a0b2-063f250ce5ad-istio-podinfo\") pod \"e2e-worker-single-main-65a246-openshift-default-7d4797c5b55m9qb\" (UID: \"de4179d8-a136-4ddd-a0b2-063f250ce5ad\") " pod="openshift-ingress/e2e-worker-single-main-65a246-openshift-default-7d4797c5b55m9qb" Nov 03 02:52:05.417867 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:05.417845 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de4179d8-a136-4ddd-a0b2-063f250ce5ad-workload-socket\") pod \"e2e-worker-single-main-65a246-openshift-default-7d4797c5b55m9qb\" (UID: \"de4179d8-a136-4ddd-a0b2-063f250ce5ad\") " pod="openshift-ingress/e2e-worker-single-main-65a246-openshift-default-7d4797c5b55m9qb" Nov 03 02:52:05.468348 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:05.468305 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:52:05.518358 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:05.518319 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de4179d8-a136-4ddd-a0b2-063f250ce5ad-istio-podinfo\") pod \"e2e-worker-single-main-65a246-openshift-default-7d4797c5b55m9qb\" (UID: \"de4179d8-a136-4ddd-a0b2-063f250ce5ad\") " pod="openshift-ingress/e2e-worker-single-main-65a246-openshift-default-7d4797c5b55m9qb" Nov 03 02:52:05.518574 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:05.518376 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de4179d8-a136-4ddd-a0b2-063f250ce5ad-workload-socket\") pod \"e2e-worker-single-main-65a246-openshift-default-7d4797c5b55m9qb\" (UID: \"de4179d8-a136-4ddd-a0b2-063f250ce5ad\") " pod="openshift-ingress/e2e-worker-single-main-65a246-openshift-default-7d4797c5b55m9qb" Nov 03 02:52:05.518574 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:05.518412 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de4179d8-a136-4ddd-a0b2-063f250ce5ad-istiod-ca-cert\") pod \"e2e-worker-single-main-65a246-openshift-default-7d4797c5b55m9qb\" (UID: \"de4179d8-a136-4ddd-a0b2-063f250ce5ad\") " pod="openshift-ingress/e2e-worker-single-main-65a246-openshift-default-7d4797c5b55m9qb" Nov 03 02:52:05.518574 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:05.518442 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de4179d8-a136-4ddd-a0b2-063f250ce5ad-istio-token\") pod \"e2e-worker-single-main-65a246-openshift-default-7d4797c5b55m9qb\" (UID: \"de4179d8-a136-4ddd-a0b2-063f250ce5ad\") " pod="openshift-ingress/e2e-worker-single-main-65a246-openshift-default-7d4797c5b55m9qb" Nov 03 02:52:05.518574 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:05.518469 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2tp7\" (UniqueName: \"kubernetes.io/projected/de4179d8-a136-4ddd-a0b2-063f250ce5ad-kube-api-access-x2tp7\") pod \"e2e-worker-single-main-65a246-openshift-default-7d4797c5b55m9qb\" (UID: \"de4179d8-a136-4ddd-a0b2-063f250ce5ad\") " pod="openshift-ingress/e2e-worker-single-main-65a246-openshift-default-7d4797c5b55m9qb" Nov 03 02:52:05.518574 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:05.518505 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de4179d8-a136-4ddd-a0b2-063f250ce5ad-istio-envoy\") pod \"e2e-worker-single-main-65a246-openshift-default-7d4797c5b55m9qb\" (UID: \"de4179d8-a136-4ddd-a0b2-063f250ce5ad\") " pod="openshift-ingress/e2e-worker-single-main-65a246-openshift-default-7d4797c5b55m9qb" Nov 03 02:52:05.518574 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:05.518555 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de4179d8-a136-4ddd-a0b2-063f250ce5ad-credential-socket\") pod \"e2e-worker-single-main-65a246-openshift-default-7d4797c5b55m9qb\" (UID: \"de4179d8-a136-4ddd-a0b2-063f250ce5ad\") " pod="openshift-ingress/e2e-worker-single-main-65a246-openshift-default-7d4797c5b55m9qb" Nov 03 02:52:05.518968 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:05.518631 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de4179d8-a136-4ddd-a0b2-063f250ce5ad-istio-data\") pod \"e2e-worker-single-main-65a246-openshift-default-7d4797c5b55m9qb\" (UID: \"de4179d8-a136-4ddd-a0b2-063f250ce5ad\") " pod="openshift-ingress/e2e-worker-single-main-65a246-openshift-default-7d4797c5b55m9qb" Nov 03 02:52:05.518968 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:05.518705 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de4179d8-a136-4ddd-a0b2-063f250ce5ad-workload-certs\") pod \"e2e-worker-single-main-65a246-openshift-default-7d4797c5b55m9qb\" (UID: \"de4179d8-a136-4ddd-a0b2-063f250ce5ad\") " pod="openshift-ingress/e2e-worker-single-main-65a246-openshift-default-7d4797c5b55m9qb" Nov 03 02:52:05.518968 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:05.518865 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de4179d8-a136-4ddd-a0b2-063f250ce5ad-workload-socket\") pod \"e2e-worker-single-main-65a246-openshift-default-7d4797c5b55m9qb\" (UID: \"de4179d8-a136-4ddd-a0b2-063f250ce5ad\") " pod="openshift-ingress/e2e-worker-single-main-65a246-openshift-default-7d4797c5b55m9qb" Nov 03 02:52:05.519142 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:05.519033 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de4179d8-a136-4ddd-a0b2-063f250ce5ad-credential-socket\") pod \"e2e-worker-single-main-65a246-openshift-default-7d4797c5b55m9qb\" (UID: \"de4179d8-a136-4ddd-a0b2-063f250ce5ad\") " pod="openshift-ingress/e2e-worker-single-main-65a246-openshift-default-7d4797c5b55m9qb" Nov 03 02:52:05.519142 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:05.519112 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de4179d8-a136-4ddd-a0b2-063f250ce5ad-istio-data\") pod \"e2e-worker-single-main-65a246-openshift-default-7d4797c5b55m9qb\" (UID: \"de4179d8-a136-4ddd-a0b2-063f250ce5ad\") " pod="openshift-ingress/e2e-worker-single-main-65a246-openshift-default-7d4797c5b55m9qb" Nov 03 02:52:05.519230 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:05.519185 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de4179d8-a136-4ddd-a0b2-063f250ce5ad-workload-certs\") pod \"e2e-worker-single-main-65a246-openshift-default-7d4797c5b55m9qb\" (UID: \"de4179d8-a136-4ddd-a0b2-063f250ce5ad\") " pod="openshift-ingress/e2e-worker-single-main-65a246-openshift-default-7d4797c5b55m9qb" Nov 03 02:52:05.519401 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:05.519354 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de4179d8-a136-4ddd-a0b2-063f250ce5ad-istiod-ca-cert\") pod \"e2e-worker-single-main-65a246-openshift-default-7d4797c5b55m9qb\" (UID: \"de4179d8-a136-4ddd-a0b2-063f250ce5ad\") " pod="openshift-ingress/e2e-worker-single-main-65a246-openshift-default-7d4797c5b55m9qb" Nov 03 02:52:05.521065 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:05.521045 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de4179d8-a136-4ddd-a0b2-063f250ce5ad-istio-podinfo\") pod \"e2e-worker-single-main-65a246-openshift-default-7d4797c5b55m9qb\" (UID: \"de4179d8-a136-4ddd-a0b2-063f250ce5ad\") " pod="openshift-ingress/e2e-worker-single-main-65a246-openshift-default-7d4797c5b55m9qb" Nov 03 02:52:05.521165 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:05.521068 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de4179d8-a136-4ddd-a0b2-063f250ce5ad-istio-envoy\") pod \"e2e-worker-single-main-65a246-openshift-default-7d4797c5b55m9qb\" (UID: \"de4179d8-a136-4ddd-a0b2-063f250ce5ad\") " pod="openshift-ingress/e2e-worker-single-main-65a246-openshift-default-7d4797c5b55m9qb" Nov 03 02:52:05.581905 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:05.581836 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de4179d8-a136-4ddd-a0b2-063f250ce5ad-istio-token\") pod \"e2e-worker-single-main-65a246-openshift-default-7d4797c5b55m9qb\" (UID: \"de4179d8-a136-4ddd-a0b2-063f250ce5ad\") " pod="openshift-ingress/e2e-worker-single-main-65a246-openshift-default-7d4797c5b55m9qb" Nov 03 02:52:05.582067 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:05.582043 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2tp7\" (UniqueName: \"kubernetes.io/projected/de4179d8-a136-4ddd-a0b2-063f250ce5ad-kube-api-access-x2tp7\") pod \"e2e-worker-single-main-65a246-openshift-default-7d4797c5b55m9qb\" (UID: \"de4179d8-a136-4ddd-a0b2-063f250ce5ad\") " pod="openshift-ingress/e2e-worker-single-main-65a246-openshift-default-7d4797c5b55m9qb" Nov 03 02:52:05.619777 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:05.619735 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-65a246-openshift-default-7d4797c5b55m9qb" Nov 03 02:52:05.878985 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:05.878905 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:52:06.038834 ip-10-0-136-16 kubenswrapper[5437]: W1103 02:52:06.038798 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde4179d8_a136_4ddd_a0b2_063f250ce5ad.slice/crio-2e6a4b30a8908c7bb3a63a0f0652d5522a8dbd43738bb5645ae0f792766e8252 WatchSource:0}: Error finding container 2e6a4b30a8908c7bb3a63a0f0652d5522a8dbd43738bb5645ae0f792766e8252: Status 404 returned error can't find the container with id 2e6a4b30a8908c7bb3a63a0f0652d5522a8dbd43738bb5645ae0f792766e8252 Nov 03 02:52:06.041107 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:06.041075 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-65a246-openshift-default-7d4797c5b55m9qb"] Nov 03 02:52:06.041654 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:06.041612 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 02:52:06.041740 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:06.041696 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 02:52:06.041792 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:06.041739 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 02:52:06.640579 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:06.640542 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-65a246-openshift-default-7d4797c5b55m9qb" event={"ID":"de4179d8-a136-4ddd-a0b2-063f250ce5ad","Type":"ContainerStarted","Data":"e4454162f654d5a4d72cce2abf2aeba674d7e7791db5c709274176241ad4d6de"} Nov 03 02:52:06.641215 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:06.640600 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-65a246-openshift-default-7d4797c5b55m9qb" event={"ID":"de4179d8-a136-4ddd-a0b2-063f250ce5ad","Type":"ContainerStarted","Data":"2e6a4b30a8908c7bb3a63a0f0652d5522a8dbd43738bb5645ae0f792766e8252"} Nov 03 02:52:06.765717 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:06.765661 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-65a246-openshift-default-7d4797c5b55m9qb" podStartSLOduration=1.765647638 podStartE2EDuration="1.765647638s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:52:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:52:06.764904705 +0000 UTC m=+5608.794386998" watchObservedRunningTime="2026-11-03 02:52:06.765647638 +0000 UTC m=+5608.795130000" Nov 03 02:52:07.620244 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:07.620201 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-65a246-openshift-default-7d4797c5b55m9qb" Nov 03 02:52:07.625547 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:07.625514 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-65a246-openshift-default-7d4797c5b55m9qb" Nov 03 02:52:07.646193 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:07.646161 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-65a246-openshift-default-7d4797c5b55m9qb" Nov 03 02:52:07.647082 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:07.647064 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-65a246-openshift-default-7d4797c5b55m9qb" Nov 03 02:52:07.759416 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:07.759356 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:52:09.972621 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:09.972561 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:52:10.199959 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:10.199784 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:52:10.428624 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:10.428581 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:52:12.719936 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:12.719893 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:52:13.197427 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:13.197389 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:52:13.464537 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:13.464444 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:52:13.529219 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:13.529180 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-65a246-65ccd96dnf"] Nov 03 02:52:13.533015 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:13.532991 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-65a246-65ccd96dnf" Nov 03 02:52:13.587666 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:13.587634 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-65a246\"" Nov 03 02:52:13.588873 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:13.588507 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-65a246-65ccd96dnf"] Nov 03 02:52:13.589038 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:13.588707 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-9591146d-dockercfg-mvnr7\"" Nov 03 02:52:13.632566 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:13.632532 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d85b5\" (UniqueName: \"kubernetes.io/projected/aca84a14-1d41-42cb-86e5-32742abb1022-kube-api-access-d85b5\") pod \"payload-pre-processing-e2e-worker-single-main-65a246-65ccd96dnf\" (UID: \"aca84a14-1d41-42cb-86e5-32742abb1022\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-65a246-65ccd96dnf" Nov 03 02:52:13.632746 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:13.632627 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aca84a14-1d41-42cb-86e5-32742abb1022-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-65a246-65ccd96dnf\" (UID: \"aca84a14-1d41-42cb-86e5-32742abb1022\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-65a246-65ccd96dnf" Nov 03 02:52:13.680558 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:13.680514 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-65a246-7448b8c55kp8zh"] Nov 03 02:52:13.684486 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:13.684460 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-65a246-7448b8c55kp8zh" Nov 03 02:52:13.733609 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:13.733519 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aca84a14-1d41-42cb-86e5-32742abb1022-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-65a246-65ccd96dnf\" (UID: \"aca84a14-1d41-42cb-86e5-32742abb1022\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-65a246-65ccd96dnf" Nov 03 02:52:13.734086 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:13.733644 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a639678f-96bf-4597-9b23-3c2f00d9cde0-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-65a246-7448b8c55kp8zh\" (UID: \"a639678f-96bf-4597-9b23-3c2f00d9cde0\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-65a246-7448b8c55kp8zh" Nov 03 02:52:13.734086 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:13.733685 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2hmb\" (UniqueName: \"kubernetes.io/projected/a639678f-96bf-4597-9b23-3c2f00d9cde0-kube-api-access-d2hmb\") pod \"payload-processing-e2e-worker-single-main-65a246-7448b8c55kp8zh\" (UID: \"a639678f-96bf-4597-9b23-3c2f00d9cde0\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-65a246-7448b8c55kp8zh" Nov 03 02:52:13.734086 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:13.733710 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d85b5\" (UniqueName: \"kubernetes.io/projected/aca84a14-1d41-42cb-86e5-32742abb1022-kube-api-access-d85b5\") pod \"payload-pre-processing-e2e-worker-single-main-65a246-65ccd96dnf\" (UID: \"aca84a14-1d41-42cb-86e5-32742abb1022\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-65a246-65ccd96dnf" Nov 03 02:52:13.734472 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:13.734444 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aca84a14-1d41-42cb-86e5-32742abb1022-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-65a246-65ccd96dnf\" (UID: \"aca84a14-1d41-42cb-86e5-32742abb1022\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-65a246-65ccd96dnf" Nov 03 02:52:13.738580 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:13.738553 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-65a246-7448b8c55kp8zh"] Nov 03 02:52:13.797102 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:13.797061 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d85b5\" (UniqueName: \"kubernetes.io/projected/aca84a14-1d41-42cb-86e5-32742abb1022-kube-api-access-d85b5\") pod \"payload-pre-processing-e2e-worker-single-main-65a246-65ccd96dnf\" (UID: \"aca84a14-1d41-42cb-86e5-32742abb1022\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-65a246-65ccd96dnf" Nov 03 02:52:13.834328 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:13.834283 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a639678f-96bf-4597-9b23-3c2f00d9cde0-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-65a246-7448b8c55kp8zh\" (UID: \"a639678f-96bf-4597-9b23-3c2f00d9cde0\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-65a246-7448b8c55kp8zh" Nov 03 02:52:13.834568 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:13.834338 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2hmb\" (UniqueName: \"kubernetes.io/projected/a639678f-96bf-4597-9b23-3c2f00d9cde0-kube-api-access-d2hmb\") pod \"payload-processing-e2e-worker-single-main-65a246-7448b8c55kp8zh\" (UID: \"a639678f-96bf-4597-9b23-3c2f00d9cde0\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-65a246-7448b8c55kp8zh" Nov 03 02:52:13.835099 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:13.835071 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a639678f-96bf-4597-9b23-3c2f00d9cde0-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-65a246-7448b8c55kp8zh\" (UID: \"a639678f-96bf-4597-9b23-3c2f00d9cde0\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-65a246-7448b8c55kp8zh" Nov 03 02:52:13.847045 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:13.847013 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-65a246-65ccd96dnf" Nov 03 02:52:13.898750 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:13.898450 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2hmb\" (UniqueName: \"kubernetes.io/projected/a639678f-96bf-4597-9b23-3c2f00d9cde0-kube-api-access-d2hmb\") pod \"payload-processing-e2e-worker-single-main-65a246-7448b8c55kp8zh\" (UID: \"a639678f-96bf-4597-9b23-3c2f00d9cde0\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-65a246-7448b8c55kp8zh" Nov 03 02:52:13.999199 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:13.999107 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-65a246-7448b8c55kp8zh" Nov 03 02:52:14.035985 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:52:14.035949 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 02:52:14.215872 ip-10-0-136-16 kubenswrapper[5437]: W1103 02:52:14.215833 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaca84a14_1d41_42cb_86e5_32742abb1022.slice/crio-ef09b418e85d820433e3b185c54fbab98c8f6d2dd2038a05b9ed7a4e8f64530c WatchSource:0}: Error finding container ef09b418e85d820433e3b185c54fbab98c8f6d2dd2038a05b9ed7a4e8f64530c: Status 404 returned error can't find the container with id ef09b418e85d820433e3b185c54fbab98c8f6d2dd2038a05b9ed7a4e8f64530c Nov 03 02:52:14.218494 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:14.218469 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-65a246-65ccd96dnf"] Nov 03 02:52:14.369988 ip-10-0-136-16 kubenswrapper[5437]: W1103 02:52:14.369962 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda639678f_96bf_4597_9b23_3c2f00d9cde0.slice/crio-535342798c23d4e3a08d4bff960003d7b4548c6aaf44363d63fcf499b5e6c12d WatchSource:0}: Error finding container 535342798c23d4e3a08d4bff960003d7b4548c6aaf44363d63fcf499b5e6c12d: Status 404 returned error can't find the container with id 535342798c23d4e3a08d4bff960003d7b4548c6aaf44363d63fcf499b5e6c12d Nov 03 02:52:14.373178 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:14.373154 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-65a246-7448b8c55kp8zh"] Nov 03 02:52:14.704576 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:14.704534 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-65a246-7448b8c55kp8zh" event={"ID":"a639678f-96bf-4597-9b23-3c2f00d9cde0","Type":"ContainerStarted","Data":"4c5eb54956e1fb8d85efde687c15ad3579c6434a5e8820df6278c62d90621fef"} Nov 03 02:52:14.704802 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:14.704636 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-65a246-7448b8c55kp8zh" Nov 03 02:52:14.704802 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:14.704662 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-65a246-7448b8c55kp8zh" event={"ID":"a639678f-96bf-4597-9b23-3c2f00d9cde0","Type":"ContainerStarted","Data":"535342798c23d4e3a08d4bff960003d7b4548c6aaf44363d63fcf499b5e6c12d"} Nov 03 02:52:14.706192 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:14.706162 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-65a246-65ccd96dnf" event={"ID":"aca84a14-1d41-42cb-86e5-32742abb1022","Type":"ContainerStarted","Data":"9ffdea93f1cd92b0d5195130ed32b6811dc1c92f30191e4c8cfeb27f3c64cf65"} Nov 03 02:52:14.706307 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:14.706216 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-65a246-65ccd96dnf" event={"ID":"aca84a14-1d41-42cb-86e5-32742abb1022","Type":"ContainerStarted","Data":"ef09b418e85d820433e3b185c54fbab98c8f6d2dd2038a05b9ed7a4e8f64530c"} Nov 03 02:52:14.706307 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:14.706247 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-65a246-65ccd96dnf" Nov 03 02:52:14.824552 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:14.824499 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-65a246-7448b8c55kp8zh" podStartSLOduration=1.8244865350000001 podStartE2EDuration="1.824486535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:52:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:52:14.824083819 +0000 UTC m=+5616.853566115" watchObservedRunningTime="2026-11-03 02:52:14.824486535 +0000 UTC m=+5616.853968829" Nov 03 02:52:14.945271 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:14.945215 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-65a246-65ccd96dnf" podStartSLOduration=1.945201357 podStartE2EDuration="1.945201357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:52:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:52:14.943396712 +0000 UTC m=+5616.972879004" watchObservedRunningTime="2026-11-03 02:52:14.945201357 +0000 UTC m=+5616.974683652" Nov 03 02:52:16.501117 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:16.501081 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-65a246-6485d574b4-l2cd4"] Nov 03 02:52:16.504977 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:16.504954 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-65a246-6485d574b4-l2cd4" Nov 03 02:52:16.560942 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:16.560909 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-65a246-6485d574b4-l2cd4"] Nov 03 02:52:16.562205 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:16.562173 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67f4556e-c9bf-4496-bdca-c567e3c273c0-metrics-certs\") pod \"maas-api-e2e-worker-single-main-65a246-6485d574b4-l2cd4\" (UID: \"67f4556e-c9bf-4496-bdca-c567e3c273c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-65a246-6485d574b4-l2cd4" Nov 03 02:52:16.562560 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:16.562534 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6n6c\" (UniqueName: \"kubernetes.io/projected/67f4556e-c9bf-4496-bdca-c567e3c273c0-kube-api-access-r6n6c\") pod \"maas-api-e2e-worker-single-main-65a246-6485d574b4-l2cd4\" (UID: \"67f4556e-c9bf-4496-bdca-c567e3c273c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-65a246-6485d574b4-l2cd4" Nov 03 02:52:16.562698 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:16.562587 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67f4556e-c9bf-4496-bdca-c567e3c273c0-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-65a246-6485d574b4-l2cd4\" (UID: \"67f4556e-c9bf-4496-bdca-c567e3c273c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-65a246-6485d574b4-l2cd4" Nov 03 02:52:16.562698 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:16.562668 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67f4556e-c9bf-4496-bdca-c567e3c273c0-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-65a246-6485d574b4-l2cd4\" (UID: \"67f4556e-c9bf-4496-bdca-c567e3c273c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-65a246-6485d574b4-l2cd4" Nov 03 02:52:16.610861 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:16.610830 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-65a246-serving-cert\"" Nov 03 02:52:16.663282 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:16.663245 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67f4556e-c9bf-4496-bdca-c567e3c273c0-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-65a246-6485d574b4-l2cd4\" (UID: \"67f4556e-c9bf-4496-bdca-c567e3c273c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-65a246-6485d574b4-l2cd4" Nov 03 02:52:16.663517 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:16.663291 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67f4556e-c9bf-4496-bdca-c567e3c273c0-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-65a246-6485d574b4-l2cd4\" (UID: \"67f4556e-c9bf-4496-bdca-c567e3c273c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-65a246-6485d574b4-l2cd4" Nov 03 02:52:16.663517 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:16.663340 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67f4556e-c9bf-4496-bdca-c567e3c273c0-metrics-certs\") pod \"maas-api-e2e-worker-single-main-65a246-6485d574b4-l2cd4\" (UID: \"67f4556e-c9bf-4496-bdca-c567e3c273c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-65a246-6485d574b4-l2cd4" Nov 03 02:52:16.663517 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:16.663427 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6n6c\" (UniqueName: \"kubernetes.io/projected/67f4556e-c9bf-4496-bdca-c567e3c273c0-kube-api-access-r6n6c\") pod \"maas-api-e2e-worker-single-main-65a246-6485d574b4-l2cd4\" (UID: \"67f4556e-c9bf-4496-bdca-c567e3c273c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-65a246-6485d574b4-l2cd4" Nov 03 02:52:16.663517 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:52:16.663439 5437 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-65a246-serving-cert: secret "maas-api-e2e-worker-single-main-65a246-serving-cert" not found Nov 03 02:52:16.663732 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:52:16.663523 5437 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67f4556e-c9bf-4496-bdca-c567e3c273c0-maas-api-tls podName:67f4556e-c9bf-4496-bdca-c567e3c273c0 nodeName:}" failed. No retries permitted until 2026-11-03 02:52:17.163494583 +0000 UTC m=+5619.192976876 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/67f4556e-c9bf-4496-bdca-c567e3c273c0-maas-api-tls") pod "maas-api-e2e-worker-single-main-65a246-6485d574b4-l2cd4" (UID: "67f4556e-c9bf-4496-bdca-c567e3c273c0") : secret "maas-api-e2e-worker-single-main-65a246-serving-cert" not found Nov 03 02:52:16.664064 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:16.664042 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67f4556e-c9bf-4496-bdca-c567e3c273c0-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-65a246-6485d574b4-l2cd4\" (UID: \"67f4556e-c9bf-4496-bdca-c567e3c273c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-65a246-6485d574b4-l2cd4" Nov 03 02:52:16.665997 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:16.665974 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67f4556e-c9bf-4496-bdca-c567e3c273c0-metrics-certs\") pod \"maas-api-e2e-worker-single-main-65a246-6485d574b4-l2cd4\" (UID: \"67f4556e-c9bf-4496-bdca-c567e3c273c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-65a246-6485d574b4-l2cd4" Nov 03 02:52:16.730460 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:16.730429 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6n6c\" (UniqueName: \"kubernetes.io/projected/67f4556e-c9bf-4496-bdca-c567e3c273c0-kube-api-access-r6n6c\") pod \"maas-api-e2e-worker-single-main-65a246-6485d574b4-l2cd4\" (UID: \"67f4556e-c9bf-4496-bdca-c567e3c273c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-65a246-6485d574b4-l2cd4" Nov 03 02:52:16.756961 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:16.756873 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:52:16.965025 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:16.964978 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:52:17.168756 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:17.168716 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67f4556e-c9bf-4496-bdca-c567e3c273c0-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-65a246-6485d574b4-l2cd4\" (UID: \"67f4556e-c9bf-4496-bdca-c567e3c273c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-65a246-6485d574b4-l2cd4" Nov 03 02:52:17.171269 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:17.171241 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67f4556e-c9bf-4496-bdca-c567e3c273c0-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-65a246-6485d574b4-l2cd4\" (UID: \"67f4556e-c9bf-4496-bdca-c567e3c273c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-65a246-6485d574b4-l2cd4" Nov 03 02:52:17.418625 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:17.418585 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-65a246-6485d574b4-l2cd4" Nov 03 02:52:17.794150 ip-10-0-136-16 kubenswrapper[5437]: W1103 02:52:17.794110 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67f4556e_c9bf_4496_bdca_c567e3c273c0.slice/crio-8b9833ee2fa78544299f1189d29dbe027a2bed713f654b7f831408663dcb5ed6 WatchSource:0}: Error finding container 8b9833ee2fa78544299f1189d29dbe027a2bed713f654b7f831408663dcb5ed6: Status 404 returned error can't find the container with id 8b9833ee2fa78544299f1189d29dbe027a2bed713f654b7f831408663dcb5ed6 Nov 03 02:52:17.797185 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:17.797152 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-65a246-6485d574b4-l2cd4"] Nov 03 02:52:18.728305 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:18.728270 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-65a246-6485d574b4-l2cd4" event={"ID":"67f4556e-c9bf-4496-bdca-c567e3c273c0","Type":"ContainerStarted","Data":"8b9833ee2fa78544299f1189d29dbe027a2bed713f654b7f831408663dcb5ed6"} Nov 03 02:52:19.735158 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:19.735114 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-65a246-6485d574b4-l2cd4" event={"ID":"67f4556e-c9bf-4496-bdca-c567e3c273c0","Type":"ContainerStarted","Data":"d75f9a7f1873850e579a41c9a83ca72b94dae146c1fe23d19892c09ebdf71452"} Nov 03 02:52:19.735582 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:19.735213 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-65a246-6485d574b4-l2cd4" Nov 03 02:52:19.856551 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:19.856498 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-65a246-6485d574b4-l2cd4" podStartSLOduration=2.836874731 podStartE2EDuration="3.856484633s" totalImagesPullingTime="1.019609902s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:52:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:52:19.855999998 +0000 UTC m=+5621.885482294" watchObservedRunningTime="2026-11-03 02:52:19.856484633 +0000 UTC m=+5621.885966922" Nov 03 02:52:22.147090 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:22.147045 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:52:25.714073 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:25.714017 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-65a246-7448b8c55kp8zh" Nov 03 02:52:25.714540 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:25.714127 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-65a246-65ccd96dnf" Nov 03 02:52:25.747746 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:25.747715 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-65a246-6485d574b4-l2cd4" Nov 03 02:52:25.766519 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:25.766475 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:52:25.984110 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:25.984028 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:52:28.588838 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:28.588798 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:52:30.796730 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:30.796691 5437 generic.go:356] "Generic (PLEG): container finished" podID="de4179d8-a136-4ddd-a0b2-063f250ce5ad" containerID="e4454162f654d5a4d72cce2abf2aeba674d7e7791db5c709274176241ad4d6de" exitCode=0 Nov 03 02:52:30.797178 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:30.796759 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-65a246-openshift-default-7d4797c5b55m9qb" event={"ID":"de4179d8-a136-4ddd-a0b2-063f250ce5ad","Type":"ContainerDied","Data":"e4454162f654d5a4d72cce2abf2aeba674d7e7791db5c709274176241ad4d6de"} Nov 03 02:52:30.797226 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:30.797189 5437 scope.go:122] "RemoveContainer" containerID="e4454162f654d5a4d72cce2abf2aeba674d7e7791db5c709274176241ad4d6de" Nov 03 02:52:30.798397 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:30.798353 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 02:52:30.798466 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:30.798444 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 02:52:30.798517 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:30.798478 5437 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 02:52:31.803318 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:31.803283 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-65a246-openshift-default-7d4797c5b55m9qb" event={"ID":"de4179d8-a136-4ddd-a0b2-063f250ce5ad","Type":"ContainerStarted","Data":"4c2b9bd8132f80174d0e2823d5ae35f8b5201d307cebbe10668c4808aa318374"} Nov 03 02:52:32.641799 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:32.641760 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-65a246-openshift-default-7d4797c5b55m9qb" Nov 03 02:52:32.642208 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:32.642110 5437 patch_prober.go:29] interesting pod/e2e-worker-single-main-65a246-openshift-default-7d4797c5b55m9qb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" start-of-body= Nov 03 02:52:32.642467 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:32.642425 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-65a246-openshift-default-7d4797c5b55m9qb" podUID="de4179d8-a136-4ddd-a0b2-063f250ce5ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" Nov 03 02:52:32.850193 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:32.850152 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:52:33.633848 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:33.633799 5437 patch_prober.go:29] interesting pod/e2e-worker-single-main-65a246-openshift-default-7d4797c5b55m9qb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" start-of-body= Nov 03 02:52:33.634147 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:33.634117 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-65a246-openshift-default-7d4797c5b55m9qb" podUID="de4179d8-a136-4ddd-a0b2-063f250ce5ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" Nov 03 02:52:34.626798 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:34.626756 5437 patch_prober.go:29] interesting pod/e2e-worker-single-main-65a246-openshift-default-7d4797c5b55m9qb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" start-of-body= Nov 03 02:52:34.627701 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:34.627665 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-65a246-openshift-default-7d4797c5b55m9qb" podUID="de4179d8-a136-4ddd-a0b2-063f250ce5ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" Nov 03 02:52:34.747068 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:34.747014 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:52:35.348566 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:35.348527 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:52:35.621010 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:35.620908 5437 patch_prober.go:29] interesting pod/e2e-worker-single-main-65a246-openshift-default-7d4797c5b55m9qb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" start-of-body= Nov 03 02:52:35.621311 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:35.621263 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-65a246-openshift-default-7d4797c5b55m9qb" podUID="de4179d8-a136-4ddd-a0b2-063f250ce5ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" Nov 03 02:52:35.654628 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:35.654591 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:52:36.624389 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:36.621820 5437 patch_prober.go:29] interesting pod/e2e-worker-single-main-65a246-openshift-default-7d4797c5b55m9qb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" start-of-body= Nov 03 02:52:36.624389 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:36.621887 5437 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-65a246-openshift-default-7d4797c5b55m9qb" podUID="de4179d8-a136-4ddd-a0b2-063f250ce5ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" Nov 03 02:52:37.624316 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:37.624283 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-65a246-openshift-default-7d4797c5b55m9qb" Nov 03 02:52:37.624938 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:37.624704 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-65a246-openshift-default-7d4797c5b55m9qb" Nov 03 02:52:37.625548 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:37.625527 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-65a246-openshift-default-7d4797c5b55m9qb" Nov 03 02:52:38.467476 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:38.467441 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:52:39.179874 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:39.179829 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:52:43.628150 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:43.628101 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:52:44.043885 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:52:44.043785 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 02:52:48.938252 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:48.938207 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:52:54.131561 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:54.131522 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:52:59.420119 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:52:59.420064 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:53:02.600107 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:53:02.600066 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:53:04.629075 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:53:04.629025 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:53:08.113067 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:53:08.113026 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:53:09.932537 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:53:09.932494 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:53:13.065642 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:53:13.065601 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:53:14.034397 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:53:14.034344 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 02:53:15.114831 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:53:15.114795 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:53:20.420618 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:53:20.420530 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:53:25.630866 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:53:25.630829 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:53:28.347984 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:53:28.347942 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:53:30.920011 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:53:30.919974 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:53:35.156803 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:53:35.156760 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:53:35.651096 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:53:35.651060 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:53:36.123763 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:53:36.123720 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:53:40.207993 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:53:40.207952 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:53:41.405506 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:53:41.405466 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:53:41.724247 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:53:41.724143 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:53:42.699240 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:53:42.699196 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-65a246-7448b8c55kp8zh"] Nov 03 02:53:42.806809 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:53:42.806775 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-65a246-65ccd96dnf"] Nov 03 02:53:43.101256 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:53:43.101207 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:53:44.044677 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:53:44.044637 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 02:53:45.142005 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:53:45.141961 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:53:46.819547 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:53:46.819505 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:53:47.533503 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:53:47.533462 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:53:47.699656 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:53:47.699603 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-65a246-7448b8c55kp8zh" podUID="a639678f-96bf-4597-9b23-3c2f00d9cde0" containerName="payload-processing" containerID="cri-o://4c5eb54956e1fb8d85efde687c15ad3579c6434a5e8820df6278c62d90621fef" gracePeriod=25 Nov 03 02:53:47.807727 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:53:47.807657 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-65a246-65ccd96dnf" podUID="aca84a14-1d41-42cb-86e5-32742abb1022" containerName="payload-pre-processing" containerID="cri-o://9ffdea93f1cd92b0d5195130ed32b6811dc1c92f30191e4c8cfeb27f3c64cf65" gracePeriod=25 Nov 03 02:53:51.252514 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:53:51.252475 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-65a246-6485d574b4-l2cd4"] Nov 03 02:53:51.252951 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:53:51.252694 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-65a246-6485d574b4-l2cd4" podUID="67f4556e-c9bf-4496-bdca-c567e3c273c0" containerName="maas-api" containerID="cri-o://d75f9a7f1873850e579a41c9a83ca72b94dae146c1fe23d19892c09ebdf71452" gracePeriod=30 Nov 03 02:53:51.413938 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:53:51.413897 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:53:52.509346 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:53:52.509316 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-65a246-6485d574b4-l2cd4" Nov 03 02:53:52.695991 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:53:52.695900 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/67f4556e-c9bf-4496-bdca-c567e3c273c0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67f4556e-c9bf-4496-bdca-c567e3c273c0-maas-api-tls\") pod \"67f4556e-c9bf-4496-bdca-c567e3c273c0\" (UID: \"67f4556e-c9bf-4496-bdca-c567e3c273c0\") " Nov 03 02:53:52.696163 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:53:52.696000 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67f4556e-c9bf-4496-bdca-c567e3c273c0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67f4556e-c9bf-4496-bdca-c567e3c273c0-openshift-service-ca\") pod \"67f4556e-c9bf-4496-bdca-c567e3c273c0\" (UID: \"67f4556e-c9bf-4496-bdca-c567e3c273c0\") " Nov 03 02:53:52.696163 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:53:52.696034 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/67f4556e-c9bf-4496-bdca-c567e3c273c0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/67f4556e-c9bf-4496-bdca-c567e3c273c0-metrics-certs\") pod \"67f4556e-c9bf-4496-bdca-c567e3c273c0\" (UID: \"67f4556e-c9bf-4496-bdca-c567e3c273c0\") " Nov 03 02:53:52.696163 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:53:52.696065 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67f4556e-c9bf-4496-bdca-c567e3c273c0-kube-api-access-r6n6c\" (UniqueName: \"kubernetes.io/projected/67f4556e-c9bf-4496-bdca-c567e3c273c0-kube-api-access-r6n6c\") pod \"67f4556e-c9bf-4496-bdca-c567e3c273c0\" (UID: \"67f4556e-c9bf-4496-bdca-c567e3c273c0\") " Nov 03 02:53:52.696433 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:53:52.696407 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67f4556e-c9bf-4496-bdca-c567e3c273c0-openshift-service-ca" pod "67f4556e-c9bf-4496-bdca-c567e3c273c0" (UID: "67f4556e-c9bf-4496-bdca-c567e3c273c0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:53:52.698347 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:53:52.698310 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67f4556e-c9bf-4496-bdca-c567e3c273c0-maas-api-tls" pod "67f4556e-c9bf-4496-bdca-c567e3c273c0" (UID: "67f4556e-c9bf-4496-bdca-c567e3c273c0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:53:52.698498 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:53:52.698382 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67f4556e-c9bf-4496-bdca-c567e3c273c0-kube-api-access-r6n6c" pod "67f4556e-c9bf-4496-bdca-c567e3c273c0" (UID: "67f4556e-c9bf-4496-bdca-c567e3c273c0"). InnerVolumeSpecName "kube-api-access-r6n6c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:53:52.698498 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:53:52.698378 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67f4556e-c9bf-4496-bdca-c567e3c273c0-metrics-certs" pod "67f4556e-c9bf-4496-bdca-c567e3c273c0" (UID: "67f4556e-c9bf-4496-bdca-c567e3c273c0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:53:52.796990 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:53:52.796951 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6n6c\" (UniqueName: \"kubernetes.io/projected/67f4556e-c9bf-4496-bdca-c567e3c273c0-kube-api-access-r6n6c\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:53:52.796990 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:53:52.796986 5437 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67f4556e-c9bf-4496-bdca-c567e3c273c0-maas-api-tls\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:53:52.796990 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:53:52.796997 5437 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67f4556e-c9bf-4496-bdca-c567e3c273c0-openshift-service-ca\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:53:52.797258 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:53:52.797006 5437 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67f4556e-c9bf-4496-bdca-c567e3c273c0-metrics-certs\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:53:53.243644 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:53:53.243602 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:53:53.252748 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:53:53.252718 5437 generic.go:356] "Generic (PLEG): container finished" podID="67f4556e-c9bf-4496-bdca-c567e3c273c0" containerID="d75f9a7f1873850e579a41c9a83ca72b94dae146c1fe23d19892c09ebdf71452" exitCode=0 Nov 03 02:53:53.252931 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:53:53.252766 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-65a246-6485d574b4-l2cd4" Nov 03 02:53:53.252931 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:53:53.252808 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-65a246-6485d574b4-l2cd4" event={"ID":"67f4556e-c9bf-4496-bdca-c567e3c273c0","Type":"ContainerDied","Data":"d75f9a7f1873850e579a41c9a83ca72b94dae146c1fe23d19892c09ebdf71452"} Nov 03 02:53:53.252931 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:53:53.252867 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-65a246-6485d574b4-l2cd4" event={"ID":"67f4556e-c9bf-4496-bdca-c567e3c273c0","Type":"ContainerDied","Data":"8b9833ee2fa78544299f1189d29dbe027a2bed713f654b7f831408663dcb5ed6"} Nov 03 02:53:53.252931 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:53:53.252895 5437 scope.go:122] "RemoveContainer" containerID="d75f9a7f1873850e579a41c9a83ca72b94dae146c1fe23d19892c09ebdf71452" Nov 03 02:53:53.265216 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:53:53.265193 5437 scope.go:122] "RemoveContainer" containerID="d75f9a7f1873850e579a41c9a83ca72b94dae146c1fe23d19892c09ebdf71452" Nov 03 02:53:53.265677 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:53:53.265655 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d75f9a7f1873850e579a41c9a83ca72b94dae146c1fe23d19892c09ebdf71452\": container with ID starting with d75f9a7f1873850e579a41c9a83ca72b94dae146c1fe23d19892c09ebdf71452 not found: ID does not exist" containerID="d75f9a7f1873850e579a41c9a83ca72b94dae146c1fe23d19892c09ebdf71452" Nov 03 02:53:53.265775 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:53:53.265690 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d75f9a7f1873850e579a41c9a83ca72b94dae146c1fe23d19892c09ebdf71452"} err="failed to get container status \"d75f9a7f1873850e579a41c9a83ca72b94dae146c1fe23d19892c09ebdf71452\": rpc error: code = NotFound desc = could not find container \"d75f9a7f1873850e579a41c9a83ca72b94dae146c1fe23d19892c09ebdf71452\": container with ID starting with d75f9a7f1873850e579a41c9a83ca72b94dae146c1fe23d19892c09ebdf71452 not found: ID does not exist" Nov 03 02:53:53.422826 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:53:53.422794 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-65a246-6485d574b4-l2cd4"] Nov 03 02:53:53.451252 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:53:53.451212 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:53:53.474411 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:53:53.474358 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-65a246-6485d574b4-l2cd4"] Nov 03 02:53:54.087738 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:53:54.087703 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67f4556e-c9bf-4496-bdca-c567e3c273c0" path="/var/lib/kubelet/pods/67f4556e-c9bf-4496-bdca-c567e3c273c0/volumes" Nov 03 02:53:55.712528 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:53:55.712491 5437 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-65a246-7448b8c55kp8zh" podUID="a639678f-96bf-4597-9b23-3c2f00d9cde0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 02:53:55.712528 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:53:55.712523 5437 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-65a246-65ccd96dnf" podUID="aca84a14-1d41-42cb-86e5-32742abb1022" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 02:54:05.712083 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:54:05.712039 5437 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-65a246-65ccd96dnf" podUID="aca84a14-1d41-42cb-86e5-32742abb1022" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 02:54:05.712532 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:54:05.712091 5437 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-65a246-7448b8c55kp8zh" podUID="a639678f-96bf-4597-9b23-3c2f00d9cde0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 02:54:07.961566 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:54:07.961494 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-65a246-7448b8c55kp8zh" Nov 03 02:54:08.052287 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:54:08.052261 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a639678f-96bf-4597-9b23-3c2f00d9cde0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a639678f-96bf-4597-9b23-3c2f00d9cde0-plugins-config-volume\") pod \"a639678f-96bf-4597-9b23-3c2f00d9cde0\" (UID: \"a639678f-96bf-4597-9b23-3c2f00d9cde0\") " Nov 03 02:54:08.052463 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:54:08.052330 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a639678f-96bf-4597-9b23-3c2f00d9cde0-kube-api-access-d2hmb\" (UniqueName: \"kubernetes.io/projected/a639678f-96bf-4597-9b23-3c2f00d9cde0-kube-api-access-d2hmb\") pod \"a639678f-96bf-4597-9b23-3c2f00d9cde0\" (UID: \"a639678f-96bf-4597-9b23-3c2f00d9cde0\") " Nov 03 02:54:08.052715 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:54:08.052663 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a639678f-96bf-4597-9b23-3c2f00d9cde0-plugins-config-volume" pod "a639678f-96bf-4597-9b23-3c2f00d9cde0" (UID: "a639678f-96bf-4597-9b23-3c2f00d9cde0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:54:08.053256 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:54:08.053237 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-65a246-65ccd96dnf" Nov 03 02:54:08.054320 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:54:08.054297 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a639678f-96bf-4597-9b23-3c2f00d9cde0-kube-api-access-d2hmb" pod "a639678f-96bf-4597-9b23-3c2f00d9cde0" (UID: "a639678f-96bf-4597-9b23-3c2f00d9cde0"). InnerVolumeSpecName "kube-api-access-d2hmb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:54:08.153880 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:54:08.153844 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aca84a14-1d41-42cb-86e5-32742abb1022-kube-api-access-d85b5\" (UniqueName: \"kubernetes.io/projected/aca84a14-1d41-42cb-86e5-32742abb1022-kube-api-access-d85b5\") pod \"aca84a14-1d41-42cb-86e5-32742abb1022\" (UID: \"aca84a14-1d41-42cb-86e5-32742abb1022\") " Nov 03 02:54:08.154061 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:54:08.153950 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aca84a14-1d41-42cb-86e5-32742abb1022-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aca84a14-1d41-42cb-86e5-32742abb1022-plugins-config-volume\") pod \"aca84a14-1d41-42cb-86e5-32742abb1022\" (UID: \"aca84a14-1d41-42cb-86e5-32742abb1022\") " Nov 03 02:54:08.154204 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:54:08.154191 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2hmb\" (UniqueName: \"kubernetes.io/projected/a639678f-96bf-4597-9b23-3c2f00d9cde0-kube-api-access-d2hmb\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:54:08.154292 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:54:08.154205 5437 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a639678f-96bf-4597-9b23-3c2f00d9cde0-plugins-config-volume\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:54:08.154347 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:54:08.154300 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aca84a14-1d41-42cb-86e5-32742abb1022-plugins-config-volume" pod "aca84a14-1d41-42cb-86e5-32742abb1022" (UID: "aca84a14-1d41-42cb-86e5-32742abb1022"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:54:08.156014 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:54:08.155991 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aca84a14-1d41-42cb-86e5-32742abb1022-kube-api-access-d85b5" pod "aca84a14-1d41-42cb-86e5-32742abb1022" (UID: "aca84a14-1d41-42cb-86e5-32742abb1022"). InnerVolumeSpecName "kube-api-access-d85b5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:54:08.254845 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:54:08.254758 5437 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aca84a14-1d41-42cb-86e5-32742abb1022-plugins-config-volume\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:54:08.254845 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:54:08.254789 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d85b5\" (UniqueName: \"kubernetes.io/projected/aca84a14-1d41-42cb-86e5-32742abb1022-kube-api-access-d85b5\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:54:08.334691 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:54:08.334645 5437 generic.go:356] "Generic (PLEG): container finished" podID="a639678f-96bf-4597-9b23-3c2f00d9cde0" containerID="4c5eb54956e1fb8d85efde687c15ad3579c6434a5e8820df6278c62d90621fef" exitCode=0 Nov 03 02:54:08.334877 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:54:08.334697 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-65a246-7448b8c55kp8zh" Nov 03 02:54:08.334877 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:54:08.334730 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-65a246-7448b8c55kp8zh" event={"ID":"a639678f-96bf-4597-9b23-3c2f00d9cde0","Type":"ContainerDied","Data":"4c5eb54956e1fb8d85efde687c15ad3579c6434a5e8820df6278c62d90621fef"} Nov 03 02:54:08.334877 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:54:08.334799 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-65a246-7448b8c55kp8zh" event={"ID":"a639678f-96bf-4597-9b23-3c2f00d9cde0","Type":"ContainerDied","Data":"535342798c23d4e3a08d4bff960003d7b4548c6aaf44363d63fcf499b5e6c12d"} Nov 03 02:54:08.334877 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:54:08.334825 5437 scope.go:122] "RemoveContainer" containerID="4c5eb54956e1fb8d85efde687c15ad3579c6434a5e8820df6278c62d90621fef" Nov 03 02:54:08.336095 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:54:08.336043 5437 generic.go:356] "Generic (PLEG): container finished" podID="aca84a14-1d41-42cb-86e5-32742abb1022" containerID="9ffdea93f1cd92b0d5195130ed32b6811dc1c92f30191e4c8cfeb27f3c64cf65" exitCode=0 Nov 03 02:54:08.336095 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:54:08.336072 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-65a246-65ccd96dnf" event={"ID":"aca84a14-1d41-42cb-86e5-32742abb1022","Type":"ContainerDied","Data":"9ffdea93f1cd92b0d5195130ed32b6811dc1c92f30191e4c8cfeb27f3c64cf65"} Nov 03 02:54:08.336221 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:54:08.336111 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-65a246-65ccd96dnf" Nov 03 02:54:08.336221 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:54:08.336116 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-65a246-65ccd96dnf" event={"ID":"aca84a14-1d41-42cb-86e5-32742abb1022","Type":"ContainerDied","Data":"ef09b418e85d820433e3b185c54fbab98c8f6d2dd2038a05b9ed7a4e8f64530c"} Nov 03 02:54:08.344165 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:54:08.344141 5437 scope.go:122] "RemoveContainer" containerID="4c5eb54956e1fb8d85efde687c15ad3579c6434a5e8820df6278c62d90621fef" Nov 03 02:54:08.344481 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:54:08.344458 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c5eb54956e1fb8d85efde687c15ad3579c6434a5e8820df6278c62d90621fef\": container with ID starting with 4c5eb54956e1fb8d85efde687c15ad3579c6434a5e8820df6278c62d90621fef not found: ID does not exist" containerID="4c5eb54956e1fb8d85efde687c15ad3579c6434a5e8820df6278c62d90621fef" Nov 03 02:54:08.344542 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:54:08.344491 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c5eb54956e1fb8d85efde687c15ad3579c6434a5e8820df6278c62d90621fef"} err="failed to get container status \"4c5eb54956e1fb8d85efde687c15ad3579c6434a5e8820df6278c62d90621fef\": rpc error: code = NotFound desc = could not find container \"4c5eb54956e1fb8d85efde687c15ad3579c6434a5e8820df6278c62d90621fef\": container with ID starting with 4c5eb54956e1fb8d85efde687c15ad3579c6434a5e8820df6278c62d90621fef not found: ID does not exist" Nov 03 02:54:08.344542 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:54:08.344510 5437 scope.go:122] "RemoveContainer" containerID="9ffdea93f1cd92b0d5195130ed32b6811dc1c92f30191e4c8cfeb27f3c64cf65" Nov 03 02:54:08.353054 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:54:08.353031 5437 scope.go:122] "RemoveContainer" containerID="9ffdea93f1cd92b0d5195130ed32b6811dc1c92f30191e4c8cfeb27f3c64cf65" Nov 03 02:54:08.353358 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:54:08.353332 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ffdea93f1cd92b0d5195130ed32b6811dc1c92f30191e4c8cfeb27f3c64cf65\": container with ID starting with 9ffdea93f1cd92b0d5195130ed32b6811dc1c92f30191e4c8cfeb27f3c64cf65 not found: ID does not exist" containerID="9ffdea93f1cd92b0d5195130ed32b6811dc1c92f30191e4c8cfeb27f3c64cf65" Nov 03 02:54:08.353447 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:54:08.353394 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ffdea93f1cd92b0d5195130ed32b6811dc1c92f30191e4c8cfeb27f3c64cf65"} err="failed to get container status \"9ffdea93f1cd92b0d5195130ed32b6811dc1c92f30191e4c8cfeb27f3c64cf65\": rpc error: code = NotFound desc = could not find container \"9ffdea93f1cd92b0d5195130ed32b6811dc1c92f30191e4c8cfeb27f3c64cf65\": container with ID starting with 9ffdea93f1cd92b0d5195130ed32b6811dc1c92f30191e4c8cfeb27f3c64cf65 not found: ID does not exist" Nov 03 02:54:08.474669 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:54:08.474634 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:54:08.505830 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:54:08.505750 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-65a246-65ccd96dnf"] Nov 03 02:54:08.510330 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:54:08.510302 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-65a246-65ccd96dnf"] Nov 03 02:54:08.623991 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:54:08.623952 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-65a246-7448b8c55kp8zh"] Nov 03 02:54:08.628653 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:54:08.628623 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-65a246-7448b8c55kp8zh"] Nov 03 02:54:10.088506 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:54:10.088464 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a639678f-96bf-4597-9b23-3c2f00d9cde0" path="/var/lib/kubelet/pods/a639678f-96bf-4597-9b23-3c2f00d9cde0/volumes" Nov 03 02:54:10.089064 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:54:10.089043 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aca84a14-1d41-42cb-86e5-32742abb1022" path="/var/lib/kubelet/pods/aca84a14-1d41-42cb-86e5-32742abb1022/volumes" Nov 03 02:54:13.424582 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:54:13.424543 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:54:14.035255 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:54:14.035216 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 02:54:17.198775 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:54:17.198741 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:54:17.468821 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:54:17.468726 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:54:21.088458 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:54:21.088418 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:54:21.746727 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:54:21.746689 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-65a246-openshift-default-7d4797c5b55m9qb"] Nov 03 02:54:21.746989 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:54:21.746958 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-65a246-openshift-default-7d4797c5b55m9qb" podUID="de4179d8-a136-4ddd-a0b2-063f250ce5ad" containerName="istio-proxy" containerID="cri-o://4c2b9bd8132f80174d0e2823d5ae35f8b5201d307cebbe10668c4808aa318374" gracePeriod=30 Nov 03 02:54:21.803621 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:54:21.803577 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:54:22.057900 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:54:22.057861 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:54:22.627438 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:54:22.627401 5437 patch_prober.go:29] interesting pod/e2e-worker-single-main-65a246-openshift-default-7d4797c5b55m9qb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 02:54:22.627880 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:54:22.627486 5437 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-65a246-openshift-default-7d4797c5b55m9qb" podUID="de4179d8-a136-4ddd-a0b2-063f250ce5ad" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 02:54:23.278058 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:54:23.278003 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:54:23.486582 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:54:23.486539 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:54:26.998536 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:54:26.998508 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-65a246-openshift-default-7d4797c5b55m9qb" Nov 03 02:54:27.142919 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:54:27.142885 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de4179d8-a136-4ddd-a0b2-063f250ce5ad-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de4179d8-a136-4ddd-a0b2-063f250ce5ad-istiod-ca-cert\") pod \"de4179d8-a136-4ddd-a0b2-063f250ce5ad\" (UID: \"de4179d8-a136-4ddd-a0b2-063f250ce5ad\") " Nov 03 02:54:27.143114 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:54:27.142989 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/de4179d8-a136-4ddd-a0b2-063f250ce5ad-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de4179d8-a136-4ddd-a0b2-063f250ce5ad-istio-podinfo\") pod \"de4179d8-a136-4ddd-a0b2-063f250ce5ad\" (UID: \"de4179d8-a136-4ddd-a0b2-063f250ce5ad\") " Nov 03 02:54:27.143114 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:54:27.143036 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de4179d8-a136-4ddd-a0b2-063f250ce5ad-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de4179d8-a136-4ddd-a0b2-063f250ce5ad-istio-data\") pod \"de4179d8-a136-4ddd-a0b2-063f250ce5ad\" (UID: \"de4179d8-a136-4ddd-a0b2-063f250ce5ad\") " Nov 03 02:54:27.143114 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:54:27.143060 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de4179d8-a136-4ddd-a0b2-063f250ce5ad-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de4179d8-a136-4ddd-a0b2-063f250ce5ad-credential-socket\") pod \"de4179d8-a136-4ddd-a0b2-063f250ce5ad\" (UID: \"de4179d8-a136-4ddd-a0b2-063f250ce5ad\") " Nov 03 02:54:27.143114 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:54:27.143081 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de4179d8-a136-4ddd-a0b2-063f250ce5ad-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de4179d8-a136-4ddd-a0b2-063f250ce5ad-workload-socket\") pod \"de4179d8-a136-4ddd-a0b2-063f250ce5ad\" (UID: \"de4179d8-a136-4ddd-a0b2-063f250ce5ad\") " Nov 03 02:54:27.143114 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:54:27.143113 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de4179d8-a136-4ddd-a0b2-063f250ce5ad-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de4179d8-a136-4ddd-a0b2-063f250ce5ad-workload-certs\") pod \"de4179d8-a136-4ddd-a0b2-063f250ce5ad\" (UID: \"de4179d8-a136-4ddd-a0b2-063f250ce5ad\") " Nov 03 02:54:27.143420 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:54:27.143144 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de4179d8-a136-4ddd-a0b2-063f250ce5ad-istio-token\" (UniqueName: \"kubernetes.io/projected/de4179d8-a136-4ddd-a0b2-063f250ce5ad-istio-token\") pod \"de4179d8-a136-4ddd-a0b2-063f250ce5ad\" (UID: \"de4179d8-a136-4ddd-a0b2-063f250ce5ad\") " Nov 03 02:54:27.143420 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:54:27.143176 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de4179d8-a136-4ddd-a0b2-063f250ce5ad-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de4179d8-a136-4ddd-a0b2-063f250ce5ad-istio-envoy\") pod \"de4179d8-a136-4ddd-a0b2-063f250ce5ad\" (UID: \"de4179d8-a136-4ddd-a0b2-063f250ce5ad\") " Nov 03 02:54:27.143420 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:54:27.143223 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de4179d8-a136-4ddd-a0b2-063f250ce5ad-kube-api-access-x2tp7\" (UniqueName: \"kubernetes.io/projected/de4179d8-a136-4ddd-a0b2-063f250ce5ad-kube-api-access-x2tp7\") pod \"de4179d8-a136-4ddd-a0b2-063f250ce5ad\" (UID: \"de4179d8-a136-4ddd-a0b2-063f250ce5ad\") " Nov 03 02:54:27.143605 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:54:27.143447 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de4179d8-a136-4ddd-a0b2-063f250ce5ad-istio-data" pod "de4179d8-a136-4ddd-a0b2-063f250ce5ad" (UID: "de4179d8-a136-4ddd-a0b2-063f250ce5ad"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:54:27.143605 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:54:27.143472 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de4179d8-a136-4ddd-a0b2-063f250ce5ad-istiod-ca-cert" pod "de4179d8-a136-4ddd-a0b2-063f250ce5ad" (UID: "de4179d8-a136-4ddd-a0b2-063f250ce5ad"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:54:27.143699 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:54:27.143628 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de4179d8-a136-4ddd-a0b2-063f250ce5ad-workload-certs" pod "de4179d8-a136-4ddd-a0b2-063f250ce5ad" (UID: "de4179d8-a136-4ddd-a0b2-063f250ce5ad"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:54:27.143793 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:54:27.143770 5437 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de4179d8-a136-4ddd-a0b2-063f250ce5ad-workload-certs\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:54:27.143851 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:54:27.143796 5437 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de4179d8-a136-4ddd-a0b2-063f250ce5ad-istiod-ca-cert\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:54:27.143851 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:54:27.143809 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de4179d8-a136-4ddd-a0b2-063f250ce5ad-credential-socket" pod "de4179d8-a136-4ddd-a0b2-063f250ce5ad" (UID: "de4179d8-a136-4ddd-a0b2-063f250ce5ad"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:54:27.143851 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:54:27.143811 5437 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de4179d8-a136-4ddd-a0b2-063f250ce5ad-istio-data\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:54:27.144019 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:54:27.143998 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de4179d8-a136-4ddd-a0b2-063f250ce5ad-workload-socket" pod "de4179d8-a136-4ddd-a0b2-063f250ce5ad" (UID: "de4179d8-a136-4ddd-a0b2-063f250ce5ad"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:54:27.145541 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:54:27.145514 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de4179d8-a136-4ddd-a0b2-063f250ce5ad-kube-api-access-x2tp7" pod "de4179d8-a136-4ddd-a0b2-063f250ce5ad" (UID: "de4179d8-a136-4ddd-a0b2-063f250ce5ad"). InnerVolumeSpecName "kube-api-access-x2tp7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:54:27.145825 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:54:27.145803 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de4179d8-a136-4ddd-a0b2-063f250ce5ad-istio-envoy" pod "de4179d8-a136-4ddd-a0b2-063f250ce5ad" (UID: "de4179d8-a136-4ddd-a0b2-063f250ce5ad"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:54:27.145825 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:54:27.145820 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/de4179d8-a136-4ddd-a0b2-063f250ce5ad-istio-podinfo" pod "de4179d8-a136-4ddd-a0b2-063f250ce5ad" (UID: "de4179d8-a136-4ddd-a0b2-063f250ce5ad"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:54:27.145953 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:54:27.145848 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de4179d8-a136-4ddd-a0b2-063f250ce5ad-istio-token" pod "de4179d8-a136-4ddd-a0b2-063f250ce5ad" (UID: "de4179d8-a136-4ddd-a0b2-063f250ce5ad"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:54:27.245147 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:54:27.245113 5437 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de4179d8-a136-4ddd-a0b2-063f250ce5ad-istio-podinfo\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:54:27.245147 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:54:27.245144 5437 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de4179d8-a136-4ddd-a0b2-063f250ce5ad-credential-socket\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:54:27.245147 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:54:27.245155 5437 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de4179d8-a136-4ddd-a0b2-063f250ce5ad-workload-socket\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:54:27.245435 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:54:27.245164 5437 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de4179d8-a136-4ddd-a0b2-063f250ce5ad-istio-token\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:54:27.245435 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:54:27.245173 5437 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de4179d8-a136-4ddd-a0b2-063f250ce5ad-istio-envoy\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:54:27.245435 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:54:27.245181 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2tp7\" (UniqueName: \"kubernetes.io/projected/de4179d8-a136-4ddd-a0b2-063f250ce5ad-kube-api-access-x2tp7\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:54:27.434380 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:54:27.434279 5437 generic.go:356] "Generic (PLEG): container finished" podID="de4179d8-a136-4ddd-a0b2-063f250ce5ad" containerID="4c2b9bd8132f80174d0e2823d5ae35f8b5201d307cebbe10668c4808aa318374" exitCode=0 Nov 03 02:54:27.434380 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:54:27.434342 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-65a246-openshift-default-7d4797c5b55m9qb" Nov 03 02:54:27.434579 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:54:27.434346 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-65a246-openshift-default-7d4797c5b55m9qb" event={"ID":"de4179d8-a136-4ddd-a0b2-063f250ce5ad","Type":"ContainerDied","Data":"4c2b9bd8132f80174d0e2823d5ae35f8b5201d307cebbe10668c4808aa318374"} Nov 03 02:54:27.434579 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:54:27.434488 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-65a246-openshift-default-7d4797c5b55m9qb" event={"ID":"de4179d8-a136-4ddd-a0b2-063f250ce5ad","Type":"ContainerDied","Data":"2e6a4b30a8908c7bb3a63a0f0652d5522a8dbd43738bb5645ae0f792766e8252"} Nov 03 02:54:27.434579 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:54:27.434515 5437 scope.go:122] "RemoveContainer" containerID="4c2b9bd8132f80174d0e2823d5ae35f8b5201d307cebbe10668c4808aa318374" Nov 03 02:54:27.443888 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:54:27.443865 5437 scope.go:122] "RemoveContainer" containerID="e4454162f654d5a4d72cce2abf2aeba674d7e7791db5c709274176241ad4d6de" Nov 03 02:54:27.452447 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:54:27.452425 5437 scope.go:122] "RemoveContainer" containerID="4c2b9bd8132f80174d0e2823d5ae35f8b5201d307cebbe10668c4808aa318374" Nov 03 02:54:27.452750 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:54:27.452728 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c2b9bd8132f80174d0e2823d5ae35f8b5201d307cebbe10668c4808aa318374\": container with ID starting with 4c2b9bd8132f80174d0e2823d5ae35f8b5201d307cebbe10668c4808aa318374 not found: ID does not exist" containerID="4c2b9bd8132f80174d0e2823d5ae35f8b5201d307cebbe10668c4808aa318374" Nov 03 02:54:27.452877 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:54:27.452761 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c2b9bd8132f80174d0e2823d5ae35f8b5201d307cebbe10668c4808aa318374"} err="failed to get container status \"4c2b9bd8132f80174d0e2823d5ae35f8b5201d307cebbe10668c4808aa318374\": rpc error: code = NotFound desc = could not find container \"4c2b9bd8132f80174d0e2823d5ae35f8b5201d307cebbe10668c4808aa318374\": container with ID starting with 4c2b9bd8132f80174d0e2823d5ae35f8b5201d307cebbe10668c4808aa318374 not found: ID does not exist" Nov 03 02:54:27.452877 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:54:27.452789 5437 scope.go:122] "RemoveContainer" containerID="e4454162f654d5a4d72cce2abf2aeba674d7e7791db5c709274176241ad4d6de" Nov 03 02:54:27.453054 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:54:27.453025 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4454162f654d5a4d72cce2abf2aeba674d7e7791db5c709274176241ad4d6de\": container with ID starting with e4454162f654d5a4d72cce2abf2aeba674d7e7791db5c709274176241ad4d6de not found: ID does not exist" containerID="e4454162f654d5a4d72cce2abf2aeba674d7e7791db5c709274176241ad4d6de" Nov 03 02:54:27.453098 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:54:27.453064 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4454162f654d5a4d72cce2abf2aeba674d7e7791db5c709274176241ad4d6de"} err="failed to get container status \"e4454162f654d5a4d72cce2abf2aeba674d7e7791db5c709274176241ad4d6de\": rpc error: code = NotFound desc = could not find container \"e4454162f654d5a4d72cce2abf2aeba674d7e7791db5c709274176241ad4d6de\": container with ID starting with e4454162f654d5a4d72cce2abf2aeba674d7e7791db5c709274176241ad4d6de not found: ID does not exist" Nov 03 02:54:27.604201 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:54:27.604168 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-65a246-openshift-default-7d4797c5b55m9qb"] Nov 03 02:54:27.609830 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:54:27.609806 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-65a246-openshift-default-7d4797c5b55m9qb"] Nov 03 02:54:28.088770 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:54:28.088732 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de4179d8-a136-4ddd-a0b2-063f250ce5ad" path="/var/lib/kubelet/pods/de4179d8-a136-4ddd-a0b2-063f250ce5ad/volumes" Nov 03 02:54:38.364117 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:54:38.364081 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-61825293-kserve-5f4ff947fc-llcdf"] Nov 03 02:54:38.364557 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:54:38.364538 5437 cpu_manager.go:414] "containerMap: removing container" podUID="aca84a14-1d41-42cb-86e5-32742abb1022" containerName="payload-pre-processing" Nov 03 02:54:38.364607 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:54:38.364560 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aca84a14-1d41-42cb-86e5-32742abb1022" containerName="payload-pre-processing" Nov 03 02:54:38.364607 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:54:38.364602 5437 cpu_manager.go:414] "containerMap: removing container" podUID="de4179d8-a136-4ddd-a0b2-063f250ce5ad" containerName="istio-proxy" Nov 03 02:54:38.364695 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:54:38.364613 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de4179d8-a136-4ddd-a0b2-063f250ce5ad" containerName="istio-proxy" Nov 03 02:54:38.364695 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:54:38.364634 5437 cpu_manager.go:414] "containerMap: removing container" podUID="67f4556e-c9bf-4496-bdca-c567e3c273c0" containerName="maas-api" Nov 03 02:54:38.364695 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:54:38.364643 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67f4556e-c9bf-4496-bdca-c567e3c273c0" containerName="maas-api" Nov 03 02:54:38.364695 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:54:38.364654 5437 cpu_manager.go:414] "containerMap: removing container" podUID="de4179d8-a136-4ddd-a0b2-063f250ce5ad" containerName="istio-proxy" Nov 03 02:54:38.364695 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:54:38.364661 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de4179d8-a136-4ddd-a0b2-063f250ce5ad" containerName="istio-proxy" Nov 03 02:54:38.364695 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:54:38.364681 5437 cpu_manager.go:414] "containerMap: removing container" podUID="a639678f-96bf-4597-9b23-3c2f00d9cde0" containerName="payload-processing" Nov 03 02:54:38.364695 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:54:38.364688 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a639678f-96bf-4597-9b23-3c2f00d9cde0" containerName="payload-processing" Nov 03 02:54:38.364959 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:54:38.364768 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="a639678f-96bf-4597-9b23-3c2f00d9cde0" containerName="payload-processing" Nov 03 02:54:38.364959 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:54:38.364777 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="aca84a14-1d41-42cb-86e5-32742abb1022" containerName="payload-pre-processing" Nov 03 02:54:38.364959 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:54:38.364787 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="de4179d8-a136-4ddd-a0b2-063f250ce5ad" containerName="istio-proxy" Nov 03 02:54:38.364959 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:54:38.364794 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="67f4556e-c9bf-4496-bdca-c567e3c273c0" containerName="maas-api" Nov 03 02:54:38.364959 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:54:38.364800 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="de4179d8-a136-4ddd-a0b2-063f250ce5ad" containerName="istio-proxy" Nov 03 02:54:38.367929 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:54:38.367911 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-61825293-kserve-5f4ff947fc-llcdf" Nov 03 02:54:38.422919 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:54:38.422887 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-61825293-kserve-5f4ff947fc-llcdf"] Nov 03 02:54:38.445026 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:54:38.444989 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4dc38652-9cb8-4b45-bbc2-6e074a8554ba-model-cache\") pod \"e2e-same-id-a-61825293-kserve-5f4ff947fc-llcdf\" (UID: \"4dc38652-9cb8-4b45-bbc2-6e074a8554ba\") " pod="llm/e2e-same-id-a-61825293-kserve-5f4ff947fc-llcdf" Nov 03 02:54:38.445026 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:54:38.445027 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4dc38652-9cb8-4b45-bbc2-6e074a8554ba-dshm\") pod \"e2e-same-id-a-61825293-kserve-5f4ff947fc-llcdf\" (UID: \"4dc38652-9cb8-4b45-bbc2-6e074a8554ba\") " pod="llm/e2e-same-id-a-61825293-kserve-5f4ff947fc-llcdf" Nov 03 02:54:38.445237 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:54:38.445167 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4dc38652-9cb8-4b45-bbc2-6e074a8554ba-tls-certs\") pod \"e2e-same-id-a-61825293-kserve-5f4ff947fc-llcdf\" (UID: \"4dc38652-9cb8-4b45-bbc2-6e074a8554ba\") " pod="llm/e2e-same-id-a-61825293-kserve-5f4ff947fc-llcdf" Nov 03 02:54:38.445237 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:54:38.445212 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4dc38652-9cb8-4b45-bbc2-6e074a8554ba-tmp-dir\") pod \"e2e-same-id-a-61825293-kserve-5f4ff947fc-llcdf\" (UID: \"4dc38652-9cb8-4b45-bbc2-6e074a8554ba\") " pod="llm/e2e-same-id-a-61825293-kserve-5f4ff947fc-llcdf" Nov 03 02:54:38.445309 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:54:38.445251 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgl76\" (UniqueName: \"kubernetes.io/projected/4dc38652-9cb8-4b45-bbc2-6e074a8554ba-kube-api-access-zgl76\") pod \"e2e-same-id-a-61825293-kserve-5f4ff947fc-llcdf\" (UID: \"4dc38652-9cb8-4b45-bbc2-6e074a8554ba\") " pod="llm/e2e-same-id-a-61825293-kserve-5f4ff947fc-llcdf" Nov 03 02:54:38.445342 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:54:38.445310 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4dc38652-9cb8-4b45-bbc2-6e074a8554ba-home\") pod \"e2e-same-id-a-61825293-kserve-5f4ff947fc-llcdf\" (UID: \"4dc38652-9cb8-4b45-bbc2-6e074a8554ba\") " pod="llm/e2e-same-id-a-61825293-kserve-5f4ff947fc-llcdf" Nov 03 02:54:38.472802 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:54:38.472766 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-61825293-kserve-self-signed-certs\"" Nov 03 02:54:38.506873 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:54:38.506836 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:54:38.546284 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:54:38.546246 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4dc38652-9cb8-4b45-bbc2-6e074a8554ba-home\") pod \"e2e-same-id-a-61825293-kserve-5f4ff947fc-llcdf\" (UID: \"4dc38652-9cb8-4b45-bbc2-6e074a8554ba\") " pod="llm/e2e-same-id-a-61825293-kserve-5f4ff947fc-llcdf" Nov 03 02:54:38.546504 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:54:38.546329 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4dc38652-9cb8-4b45-bbc2-6e074a8554ba-model-cache\") pod \"e2e-same-id-a-61825293-kserve-5f4ff947fc-llcdf\" (UID: \"4dc38652-9cb8-4b45-bbc2-6e074a8554ba\") " pod="llm/e2e-same-id-a-61825293-kserve-5f4ff947fc-llcdf" Nov 03 02:54:38.546504 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:54:38.546360 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4dc38652-9cb8-4b45-bbc2-6e074a8554ba-dshm\") pod \"e2e-same-id-a-61825293-kserve-5f4ff947fc-llcdf\" (UID: \"4dc38652-9cb8-4b45-bbc2-6e074a8554ba\") " pod="llm/e2e-same-id-a-61825293-kserve-5f4ff947fc-llcdf" Nov 03 02:54:38.546504 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:54:38.546451 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4dc38652-9cb8-4b45-bbc2-6e074a8554ba-tls-certs\") pod \"e2e-same-id-a-61825293-kserve-5f4ff947fc-llcdf\" (UID: \"4dc38652-9cb8-4b45-bbc2-6e074a8554ba\") " pod="llm/e2e-same-id-a-61825293-kserve-5f4ff947fc-llcdf" Nov 03 02:54:38.546504 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:54:38.546474 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4dc38652-9cb8-4b45-bbc2-6e074a8554ba-tmp-dir\") pod \"e2e-same-id-a-61825293-kserve-5f4ff947fc-llcdf\" (UID: \"4dc38652-9cb8-4b45-bbc2-6e074a8554ba\") " pod="llm/e2e-same-id-a-61825293-kserve-5f4ff947fc-llcdf" Nov 03 02:54:38.546731 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:54:38.546505 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgl76\" (UniqueName: \"kubernetes.io/projected/4dc38652-9cb8-4b45-bbc2-6e074a8554ba-kube-api-access-zgl76\") pod \"e2e-same-id-a-61825293-kserve-5f4ff947fc-llcdf\" (UID: \"4dc38652-9cb8-4b45-bbc2-6e074a8554ba\") " pod="llm/e2e-same-id-a-61825293-kserve-5f4ff947fc-llcdf" Nov 03 02:54:38.546731 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:54:38.546664 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4dc38652-9cb8-4b45-bbc2-6e074a8554ba-home\") pod \"e2e-same-id-a-61825293-kserve-5f4ff947fc-llcdf\" (UID: \"4dc38652-9cb8-4b45-bbc2-6e074a8554ba\") " pod="llm/e2e-same-id-a-61825293-kserve-5f4ff947fc-llcdf" Nov 03 02:54:38.546731 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:54:38.546691 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4dc38652-9cb8-4b45-bbc2-6e074a8554ba-model-cache\") pod \"e2e-same-id-a-61825293-kserve-5f4ff947fc-llcdf\" (UID: \"4dc38652-9cb8-4b45-bbc2-6e074a8554ba\") " pod="llm/e2e-same-id-a-61825293-kserve-5f4ff947fc-llcdf" Nov 03 02:54:38.546884 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:54:38.546854 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4dc38652-9cb8-4b45-bbc2-6e074a8554ba-tmp-dir\") pod \"e2e-same-id-a-61825293-kserve-5f4ff947fc-llcdf\" (UID: \"4dc38652-9cb8-4b45-bbc2-6e074a8554ba\") " pod="llm/e2e-same-id-a-61825293-kserve-5f4ff947fc-llcdf" Nov 03 02:54:38.548522 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:54:38.548489 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4dc38652-9cb8-4b45-bbc2-6e074a8554ba-dshm\") pod \"e2e-same-id-a-61825293-kserve-5f4ff947fc-llcdf\" (UID: \"4dc38652-9cb8-4b45-bbc2-6e074a8554ba\") " pod="llm/e2e-same-id-a-61825293-kserve-5f4ff947fc-llcdf" Nov 03 02:54:38.548749 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:54:38.548730 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4dc38652-9cb8-4b45-bbc2-6e074a8554ba-tls-certs\") pod \"e2e-same-id-a-61825293-kserve-5f4ff947fc-llcdf\" (UID: \"4dc38652-9cb8-4b45-bbc2-6e074a8554ba\") " pod="llm/e2e-same-id-a-61825293-kserve-5f4ff947fc-llcdf" Nov 03 02:54:38.610562 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:54:38.610528 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgl76\" (UniqueName: \"kubernetes.io/projected/4dc38652-9cb8-4b45-bbc2-6e074a8554ba-kube-api-access-zgl76\") pod \"e2e-same-id-a-61825293-kserve-5f4ff947fc-llcdf\" (UID: \"4dc38652-9cb8-4b45-bbc2-6e074a8554ba\") " pod="llm/e2e-same-id-a-61825293-kserve-5f4ff947fc-llcdf" Nov 03 02:54:38.680333 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:54:38.680247 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-61825293-kserve-5f4ff947fc-llcdf" Nov 03 02:54:39.088706 ip-10-0-136-16 kubenswrapper[5437]: W1103 02:54:39.088671 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4dc38652_9cb8_4b45_bbc2_6e074a8554ba.slice/crio-d23254009ccf73629141f88231e3299bdca993f184cea873e34e924bef0bd548 WatchSource:0}: Error finding container d23254009ccf73629141f88231e3299bdca993f184cea873e34e924bef0bd548: Status 404 returned error can't find the container with id d23254009ccf73629141f88231e3299bdca993f184cea873e34e924bef0bd548 Nov 03 02:54:39.090245 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:54:39.090220 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-61825293-kserve-5f4ff947fc-llcdf"] Nov 03 02:54:39.505121 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:54:39.505006 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-61825293-kserve-5f4ff947fc-llcdf" event={"ID":"4dc38652-9cb8-4b45-bbc2-6e074a8554ba","Type":"ContainerStarted","Data":"454ad8bd07f7dad76257b8c7049df62627cf2562e7af45f01cdecb9f15e7452f"} Nov 03 02:54:39.505121 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:54:39.505078 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-61825293-kserve-5f4ff947fc-llcdf" event={"ID":"4dc38652-9cb8-4b45-bbc2-6e074a8554ba","Type":"ContainerStarted","Data":"d23254009ccf73629141f88231e3299bdca993f184cea873e34e924bef0bd548"} Nov 03 02:54:39.626911 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:54:39.626846 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-61825293-kserve-5f4ff947fc-llcdf" podStartSLOduration=1.626831661 podStartE2EDuration="1.626831661s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:54:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:54:39.625446453 +0000 UTC m=+5761.654928748" watchObservedRunningTime="2026-11-03 02:54:39.626831661 +0000 UTC m=+5761.656313956" Nov 03 02:54:39.680460 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:54:39.680422 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-61825293-kserve-5f4ff947fc-llcdf" Nov 03 02:54:44.040260 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:54:44.040211 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 02:54:48.415456 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:54:48.415349 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:54:48.622660 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:54:48.622605 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:54:48.680819 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:54:48.680736 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-61825293-kserve-5f4ff947fc-llcdf" Nov 03 02:54:48.692966 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:54:48.692935 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-61825293-kserve-5f4ff947fc-llcdf" Nov 03 02:54:49.583778 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:54:49.583747 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-61825293-kserve-5f4ff947fc-llcdf" Nov 03 02:55:04.966588 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:55:04.966551 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-61825293-kserve-55f875484f-tndd6"] Nov 03 02:55:04.970093 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:55:04.970072 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-61825293-kserve-55f875484f-tndd6" Nov 03 02:55:05.025571 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:55:05.025536 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-61825293-kserve-55f875484f-tndd6"] Nov 03 02:55:05.075418 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:55:05.075348 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-61825293-kserve-self-signed-certs\"" Nov 03 02:55:05.131405 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:55:05.131350 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:55:05.134590 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:55:05.134561 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e45b229c-4bb3-4736-a126-af6c6484f54c-tls-certs\") pod \"e2e-same-id-b-61825293-kserve-55f875484f-tndd6\" (UID: \"e45b229c-4bb3-4736-a126-af6c6484f54c\") " pod="llm/e2e-same-id-b-61825293-kserve-55f875484f-tndd6" Nov 03 02:55:05.134744 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:55:05.134623 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e45b229c-4bb3-4736-a126-af6c6484f54c-tmp-dir\") pod \"e2e-same-id-b-61825293-kserve-55f875484f-tndd6\" (UID: \"e45b229c-4bb3-4736-a126-af6c6484f54c\") " pod="llm/e2e-same-id-b-61825293-kserve-55f875484f-tndd6" Nov 03 02:55:05.134804 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:55:05.134770 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e45b229c-4bb3-4736-a126-af6c6484f54c-home\") pod \"e2e-same-id-b-61825293-kserve-55f875484f-tndd6\" (UID: \"e45b229c-4bb3-4736-a126-af6c6484f54c\") " pod="llm/e2e-same-id-b-61825293-kserve-55f875484f-tndd6" Nov 03 02:55:05.134891 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:55:05.134861 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e45b229c-4bb3-4736-a126-af6c6484f54c-dshm\") pod \"e2e-same-id-b-61825293-kserve-55f875484f-tndd6\" (UID: \"e45b229c-4bb3-4736-a126-af6c6484f54c\") " pod="llm/e2e-same-id-b-61825293-kserve-55f875484f-tndd6" Nov 03 02:55:05.134953 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:55:05.134896 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e45b229c-4bb3-4736-a126-af6c6484f54c-model-cache\") pod \"e2e-same-id-b-61825293-kserve-55f875484f-tndd6\" (UID: \"e45b229c-4bb3-4736-a126-af6c6484f54c\") " pod="llm/e2e-same-id-b-61825293-kserve-55f875484f-tndd6" Nov 03 02:55:05.135049 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:55:05.135032 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z42r2\" (UniqueName: \"kubernetes.io/projected/e45b229c-4bb3-4736-a126-af6c6484f54c-kube-api-access-z42r2\") pod \"e2e-same-id-b-61825293-kserve-55f875484f-tndd6\" (UID: \"e45b229c-4bb3-4736-a126-af6c6484f54c\") " pod="llm/e2e-same-id-b-61825293-kserve-55f875484f-tndd6" Nov 03 02:55:05.235929 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:55:05.235830 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e45b229c-4bb3-4736-a126-af6c6484f54c-dshm\") pod \"e2e-same-id-b-61825293-kserve-55f875484f-tndd6\" (UID: \"e45b229c-4bb3-4736-a126-af6c6484f54c\") " pod="llm/e2e-same-id-b-61825293-kserve-55f875484f-tndd6" Nov 03 02:55:05.235929 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:55:05.235874 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e45b229c-4bb3-4736-a126-af6c6484f54c-model-cache\") pod \"e2e-same-id-b-61825293-kserve-55f875484f-tndd6\" (UID: \"e45b229c-4bb3-4736-a126-af6c6484f54c\") " pod="llm/e2e-same-id-b-61825293-kserve-55f875484f-tndd6" Nov 03 02:55:05.235929 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:55:05.235896 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z42r2\" (UniqueName: \"kubernetes.io/projected/e45b229c-4bb3-4736-a126-af6c6484f54c-kube-api-access-z42r2\") pod \"e2e-same-id-b-61825293-kserve-55f875484f-tndd6\" (UID: \"e45b229c-4bb3-4736-a126-af6c6484f54c\") " pod="llm/e2e-same-id-b-61825293-kserve-55f875484f-tndd6" Nov 03 02:55:05.236225 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:55:05.235933 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e45b229c-4bb3-4736-a126-af6c6484f54c-tls-certs\") pod \"e2e-same-id-b-61825293-kserve-55f875484f-tndd6\" (UID: \"e45b229c-4bb3-4736-a126-af6c6484f54c\") " pod="llm/e2e-same-id-b-61825293-kserve-55f875484f-tndd6" Nov 03 02:55:05.236225 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:55:05.235981 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e45b229c-4bb3-4736-a126-af6c6484f54c-tmp-dir\") pod \"e2e-same-id-b-61825293-kserve-55f875484f-tndd6\" (UID: \"e45b229c-4bb3-4736-a126-af6c6484f54c\") " pod="llm/e2e-same-id-b-61825293-kserve-55f875484f-tndd6" Nov 03 02:55:05.236225 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:55:05.236037 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e45b229c-4bb3-4736-a126-af6c6484f54c-home\") pod \"e2e-same-id-b-61825293-kserve-55f875484f-tndd6\" (UID: \"e45b229c-4bb3-4736-a126-af6c6484f54c\") " pod="llm/e2e-same-id-b-61825293-kserve-55f875484f-tndd6" Nov 03 02:55:05.236397 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:55:05.236341 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e45b229c-4bb3-4736-a126-af6c6484f54c-model-cache\") pod \"e2e-same-id-b-61825293-kserve-55f875484f-tndd6\" (UID: \"e45b229c-4bb3-4736-a126-af6c6484f54c\") " pod="llm/e2e-same-id-b-61825293-kserve-55f875484f-tndd6" Nov 03 02:55:05.236469 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:55:05.236439 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e45b229c-4bb3-4736-a126-af6c6484f54c-tmp-dir\") pod \"e2e-same-id-b-61825293-kserve-55f875484f-tndd6\" (UID: \"e45b229c-4bb3-4736-a126-af6c6484f54c\") " pod="llm/e2e-same-id-b-61825293-kserve-55f875484f-tndd6" Nov 03 02:55:05.236469 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:55:05.236448 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e45b229c-4bb3-4736-a126-af6c6484f54c-home\") pod \"e2e-same-id-b-61825293-kserve-55f875484f-tndd6\" (UID: \"e45b229c-4bb3-4736-a126-af6c6484f54c\") " pod="llm/e2e-same-id-b-61825293-kserve-55f875484f-tndd6" Nov 03 02:55:05.238231 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:55:05.238199 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e45b229c-4bb3-4736-a126-af6c6484f54c-dshm\") pod \"e2e-same-id-b-61825293-kserve-55f875484f-tndd6\" (UID: \"e45b229c-4bb3-4736-a126-af6c6484f54c\") " pod="llm/e2e-same-id-b-61825293-kserve-55f875484f-tndd6" Nov 03 02:55:05.238649 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:55:05.238627 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e45b229c-4bb3-4736-a126-af6c6484f54c-tls-certs\") pod \"e2e-same-id-b-61825293-kserve-55f875484f-tndd6\" (UID: \"e45b229c-4bb3-4736-a126-af6c6484f54c\") " pod="llm/e2e-same-id-b-61825293-kserve-55f875484f-tndd6" Nov 03 02:55:05.298823 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:55:05.298793 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z42r2\" (UniqueName: \"kubernetes.io/projected/e45b229c-4bb3-4736-a126-af6c6484f54c-kube-api-access-z42r2\") pod \"e2e-same-id-b-61825293-kserve-55f875484f-tndd6\" (UID: \"e45b229c-4bb3-4736-a126-af6c6484f54c\") " pod="llm/e2e-same-id-b-61825293-kserve-55f875484f-tndd6" Nov 03 02:55:05.582208 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:55:05.582174 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-61825293-kserve-55f875484f-tndd6" Nov 03 02:55:05.952572 ip-10-0-136-16 kubenswrapper[5437]: W1103 02:55:05.952537 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode45b229c_4bb3_4736_a126_af6c6484f54c.slice/crio-daae2e337ebe9f59f5fcfba611fd9e30afa17f66a28116d32bbb3edcec677e86 WatchSource:0}: Error finding container daae2e337ebe9f59f5fcfba611fd9e30afa17f66a28116d32bbb3edcec677e86: Status 404 returned error can't find the container with id daae2e337ebe9f59f5fcfba611fd9e30afa17f66a28116d32bbb3edcec677e86 Nov 03 02:55:05.954008 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:55:05.953982 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-61825293-kserve-55f875484f-tndd6"] Nov 03 02:55:06.661339 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:55:06.661302 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-61825293-kserve-55f875484f-tndd6" event={"ID":"e45b229c-4bb3-4736-a126-af6c6484f54c","Type":"ContainerStarted","Data":"1139327ab10b4ca90dc4a09df976bf06b4fd2c4d7327033eaf7fee7b0f5d6367"} Nov 03 02:55:06.661796 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:55:06.661346 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-61825293-kserve-55f875484f-tndd6" event={"ID":"e45b229c-4bb3-4736-a126-af6c6484f54c","Type":"ContainerStarted","Data":"daae2e337ebe9f59f5fcfba611fd9e30afa17f66a28116d32bbb3edcec677e86"} Nov 03 02:55:06.781900 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:55:06.781841 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-61825293-kserve-55f875484f-tndd6" podStartSLOduration=2.7818288 podStartE2EDuration="2.7818288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:55:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:55:06.781304078 +0000 UTC m=+5788.810786373" watchObservedRunningTime="2026-11-03 02:55:06.7818288 +0000 UTC m=+5788.811311095" Nov 03 02:55:07.582530 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:55:07.582487 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-61825293-kserve-55f875484f-tndd6" Nov 03 02:55:08.875060 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:55:08.875018 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:55:12.881389 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:55:12.880487 5437 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-x2xwt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 02:55:12.881389 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:55:12.880560 5437 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-x2xwt" podUID="f9cb50c6-d76d-4138-8d00-6b7836297ed7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 02:55:13.779213 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:55:13.779167 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:55:14.033727 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:55:14.033647 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 02:55:15.582988 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:55:15.582949 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-61825293-kserve-55f875484f-tndd6" Nov 03 02:55:15.595583 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:55:15.595553 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-61825293-kserve-55f875484f-tndd6" Nov 03 02:55:15.722232 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:55:15.722202 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-61825293-kserve-55f875484f-tndd6" Nov 03 02:55:44.034226 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:55:44.034183 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 02:55:44.187134 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:55:44.187090 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-61825293-kserve-5f4ff947fc-llcdf"] Nov 03 02:55:44.266601 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:55:44.266567 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:55:51.226793 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:55:51.226756 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-61825293-kserve-55f875484f-tndd6"] Nov 03 02:55:51.295386 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:55:51.295326 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:55:59.185595 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:55:59.185510 5437 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-zj6mx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 02:55:59.186299 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:55:59.186241 5437 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-zj6mx" podUID="fd1010b6-bf5b-4633-a266-9b5dcf9b1963" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 02:55:59.207507 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:55:59.207462 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-61825293-kserve-5f4ff947fc-llcdf" podUID="4dc38652-9cb8-4b45-bbc2-6e074a8554ba" containerName="main" containerID="cri-o://454ad8bd07f7dad76257b8c7049df62627cf2562e7af45f01cdecb9f15e7452f" gracePeriod=45 Nov 03 02:55:59.503570 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:55:59.503539 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-61825293-kserve-5f4ff947fc-llcdf" Nov 03 02:55:59.604613 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:55:59.604580 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4dc38652-9cb8-4b45-bbc2-6e074a8554ba-kube-api-access-zgl76\" (UniqueName: \"kubernetes.io/projected/4dc38652-9cb8-4b45-bbc2-6e074a8554ba-kube-api-access-zgl76\") pod \"4dc38652-9cb8-4b45-bbc2-6e074a8554ba\" (UID: \"4dc38652-9cb8-4b45-bbc2-6e074a8554ba\") " Nov 03 02:55:59.604834 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:55:59.604641 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4dc38652-9cb8-4b45-bbc2-6e074a8554ba-home\" (UniqueName: \"kubernetes.io/empty-dir/4dc38652-9cb8-4b45-bbc2-6e074a8554ba-home\") pod \"4dc38652-9cb8-4b45-bbc2-6e074a8554ba\" (UID: \"4dc38652-9cb8-4b45-bbc2-6e074a8554ba\") " Nov 03 02:55:59.604834 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:55:59.604666 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4dc38652-9cb8-4b45-bbc2-6e074a8554ba-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4dc38652-9cb8-4b45-bbc2-6e074a8554ba-tmp-dir\") pod \"4dc38652-9cb8-4b45-bbc2-6e074a8554ba\" (UID: \"4dc38652-9cb8-4b45-bbc2-6e074a8554ba\") " Nov 03 02:55:59.604834 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:55:59.604683 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4dc38652-9cb8-4b45-bbc2-6e074a8554ba-tls-certs\" (UniqueName: \"kubernetes.io/secret/4dc38652-9cb8-4b45-bbc2-6e074a8554ba-tls-certs\") pod \"4dc38652-9cb8-4b45-bbc2-6e074a8554ba\" (UID: \"4dc38652-9cb8-4b45-bbc2-6e074a8554ba\") " Nov 03 02:55:59.604834 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:55:59.604742 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4dc38652-9cb8-4b45-bbc2-6e074a8554ba-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4dc38652-9cb8-4b45-bbc2-6e074a8554ba-dshm\") pod \"4dc38652-9cb8-4b45-bbc2-6e074a8554ba\" (UID: \"4dc38652-9cb8-4b45-bbc2-6e074a8554ba\") " Nov 03 02:55:59.604834 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:55:59.604780 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4dc38652-9cb8-4b45-bbc2-6e074a8554ba-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4dc38652-9cb8-4b45-bbc2-6e074a8554ba-model-cache\") pod \"4dc38652-9cb8-4b45-bbc2-6e074a8554ba\" (UID: \"4dc38652-9cb8-4b45-bbc2-6e074a8554ba\") " Nov 03 02:55:59.605135 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:55:59.605006 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dc38652-9cb8-4b45-bbc2-6e074a8554ba-home" pod "4dc38652-9cb8-4b45-bbc2-6e074a8554ba" (UID: "4dc38652-9cb8-4b45-bbc2-6e074a8554ba"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:55:59.605135 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:55:59.605058 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dc38652-9cb8-4b45-bbc2-6e074a8554ba-tmp-dir" pod "4dc38652-9cb8-4b45-bbc2-6e074a8554ba" (UID: "4dc38652-9cb8-4b45-bbc2-6e074a8554ba"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:55:59.605247 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:55:59.605170 5437 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4dc38652-9cb8-4b45-bbc2-6e074a8554ba-home\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:55:59.605247 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:55:59.605174 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dc38652-9cb8-4b45-bbc2-6e074a8554ba-model-cache" pod "4dc38652-9cb8-4b45-bbc2-6e074a8554ba" (UID: "4dc38652-9cb8-4b45-bbc2-6e074a8554ba"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:55:59.605247 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:55:59.605184 5437 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4dc38652-9cb8-4b45-bbc2-6e074a8554ba-tmp-dir\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:55:59.606927 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:55:59.606895 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4dc38652-9cb8-4b45-bbc2-6e074a8554ba-kube-api-access-zgl76" pod "4dc38652-9cb8-4b45-bbc2-6e074a8554ba" (UID: "4dc38652-9cb8-4b45-bbc2-6e074a8554ba"). InnerVolumeSpecName "kube-api-access-zgl76". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:55:59.607055 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:55:59.606991 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4dc38652-9cb8-4b45-bbc2-6e074a8554ba-tls-certs" pod "4dc38652-9cb8-4b45-bbc2-6e074a8554ba" (UID: "4dc38652-9cb8-4b45-bbc2-6e074a8554ba"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:55:59.607055 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:55:59.606998 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dc38652-9cb8-4b45-bbc2-6e074a8554ba-dshm" pod "4dc38652-9cb8-4b45-bbc2-6e074a8554ba" (UID: "4dc38652-9cb8-4b45-bbc2-6e074a8554ba"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:55:59.706158 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:55:59.706119 5437 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4dc38652-9cb8-4b45-bbc2-6e074a8554ba-tls-certs\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:55:59.706158 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:55:59.706159 5437 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4dc38652-9cb8-4b45-bbc2-6e074a8554ba-dshm\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:55:59.706400 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:55:59.706173 5437 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4dc38652-9cb8-4b45-bbc2-6e074a8554ba-model-cache\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:55:59.706400 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:55:59.706189 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgl76\" (UniqueName: \"kubernetes.io/projected/4dc38652-9cb8-4b45-bbc2-6e074a8554ba-kube-api-access-zgl76\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:55:59.946885 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:55:59.946852 5437 generic.go:356] "Generic (PLEG): container finished" podID="4dc38652-9cb8-4b45-bbc2-6e074a8554ba" containerID="454ad8bd07f7dad76257b8c7049df62627cf2562e7af45f01cdecb9f15e7452f" exitCode=0 Nov 03 02:55:59.947089 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:55:59.946897 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-61825293-kserve-5f4ff947fc-llcdf" Nov 03 02:55:59.947089 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:55:59.946938 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-61825293-kserve-5f4ff947fc-llcdf" event={"ID":"4dc38652-9cb8-4b45-bbc2-6e074a8554ba","Type":"ContainerDied","Data":"454ad8bd07f7dad76257b8c7049df62627cf2562e7af45f01cdecb9f15e7452f"} Nov 03 02:55:59.947089 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:55:59.946979 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-61825293-kserve-5f4ff947fc-llcdf" event={"ID":"4dc38652-9cb8-4b45-bbc2-6e074a8554ba","Type":"ContainerDied","Data":"d23254009ccf73629141f88231e3299bdca993f184cea873e34e924bef0bd548"} Nov 03 02:55:59.947089 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:55:59.946996 5437 scope.go:122] "RemoveContainer" containerID="454ad8bd07f7dad76257b8c7049df62627cf2562e7af45f01cdecb9f15e7452f" Nov 03 02:55:59.956767 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:55:59.956750 5437 scope.go:122] "RemoveContainer" containerID="454ad8bd07f7dad76257b8c7049df62627cf2562e7af45f01cdecb9f15e7452f" Nov 03 02:55:59.957065 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:55:59.957044 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"454ad8bd07f7dad76257b8c7049df62627cf2562e7af45f01cdecb9f15e7452f\": container with ID starting with 454ad8bd07f7dad76257b8c7049df62627cf2562e7af45f01cdecb9f15e7452f not found: ID does not exist" containerID="454ad8bd07f7dad76257b8c7049df62627cf2562e7af45f01cdecb9f15e7452f" Nov 03 02:55:59.957111 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:55:59.957075 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"454ad8bd07f7dad76257b8c7049df62627cf2562e7af45f01cdecb9f15e7452f"} err="failed to get container status \"454ad8bd07f7dad76257b8c7049df62627cf2562e7af45f01cdecb9f15e7452f\": rpc error: code = NotFound desc = could not find container \"454ad8bd07f7dad76257b8c7049df62627cf2562e7af45f01cdecb9f15e7452f\": container with ID starting with 454ad8bd07f7dad76257b8c7049df62627cf2562e7af45f01cdecb9f15e7452f not found: ID does not exist" Nov 03 02:56:00.117709 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:56:00.117669 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-61825293-kserve-5f4ff947fc-llcdf"] Nov 03 02:56:00.169073 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:56:00.169023 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-61825293-kserve-5f4ff947fc-llcdf"] Nov 03 02:56:02.087543 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:56:02.087508 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4dc38652-9cb8-4b45-bbc2-6e074a8554ba" path="/var/lib/kubelet/pods/4dc38652-9cb8-4b45-bbc2-6e074a8554ba/volumes" Nov 03 02:56:06.246557 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:56:06.246510 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-61825293-kserve-55f875484f-tndd6" podUID="e45b229c-4bb3-4736-a126-af6c6484f54c" containerName="main" containerID="cri-o://1139327ab10b4ca90dc4a09df976bf06b4fd2c4d7327033eaf7fee7b0f5d6367" gracePeriod=45 Nov 03 02:56:06.501276 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:56:06.501203 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-61825293-kserve-55f875484f-tndd6" Nov 03 02:56:06.571126 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:56:06.571090 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e45b229c-4bb3-4736-a126-af6c6484f54c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e45b229c-4bb3-4736-a126-af6c6484f54c-dshm\") pod \"e45b229c-4bb3-4736-a126-af6c6484f54c\" (UID: \"e45b229c-4bb3-4736-a126-af6c6484f54c\") " Nov 03 02:56:06.571300 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:56:06.571161 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e45b229c-4bb3-4736-a126-af6c6484f54c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e45b229c-4bb3-4736-a126-af6c6484f54c-tmp-dir\") pod \"e45b229c-4bb3-4736-a126-af6c6484f54c\" (UID: \"e45b229c-4bb3-4736-a126-af6c6484f54c\") " Nov 03 02:56:06.571300 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:56:06.571186 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e45b229c-4bb3-4736-a126-af6c6484f54c-tls-certs\" (UniqueName: \"kubernetes.io/secret/e45b229c-4bb3-4736-a126-af6c6484f54c-tls-certs\") pod \"e45b229c-4bb3-4736-a126-af6c6484f54c\" (UID: \"e45b229c-4bb3-4736-a126-af6c6484f54c\") " Nov 03 02:56:06.571300 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:56:06.571222 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e45b229c-4bb3-4736-a126-af6c6484f54c-home\" (UniqueName: \"kubernetes.io/empty-dir/e45b229c-4bb3-4736-a126-af6c6484f54c-home\") pod \"e45b229c-4bb3-4736-a126-af6c6484f54c\" (UID: \"e45b229c-4bb3-4736-a126-af6c6484f54c\") " Nov 03 02:56:06.571300 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:56:06.571276 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e45b229c-4bb3-4736-a126-af6c6484f54c-kube-api-access-z42r2\" (UniqueName: \"kubernetes.io/projected/e45b229c-4bb3-4736-a126-af6c6484f54c-kube-api-access-z42r2\") pod \"e45b229c-4bb3-4736-a126-af6c6484f54c\" (UID: \"e45b229c-4bb3-4736-a126-af6c6484f54c\") " Nov 03 02:56:06.571553 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:56:06.571326 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e45b229c-4bb3-4736-a126-af6c6484f54c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e45b229c-4bb3-4736-a126-af6c6484f54c-model-cache\") pod \"e45b229c-4bb3-4736-a126-af6c6484f54c\" (UID: \"e45b229c-4bb3-4736-a126-af6c6484f54c\") " Nov 03 02:56:06.571615 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:56:06.571538 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e45b229c-4bb3-4736-a126-af6c6484f54c-tmp-dir" pod "e45b229c-4bb3-4736-a126-af6c6484f54c" (UID: "e45b229c-4bb3-4736-a126-af6c6484f54c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:56:06.571615 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:56:06.571575 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e45b229c-4bb3-4736-a126-af6c6484f54c-home" pod "e45b229c-4bb3-4736-a126-af6c6484f54c" (UID: "e45b229c-4bb3-4736-a126-af6c6484f54c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:56:06.571797 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:56:06.571768 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e45b229c-4bb3-4736-a126-af6c6484f54c-model-cache" pod "e45b229c-4bb3-4736-a126-af6c6484f54c" (UID: "e45b229c-4bb3-4736-a126-af6c6484f54c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:56:06.571933 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:56:06.571868 5437 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e45b229c-4bb3-4736-a126-af6c6484f54c-tmp-dir\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:56:06.571933 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:56:06.571890 5437 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e45b229c-4bb3-4736-a126-af6c6484f54c-home\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:56:06.573511 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:56:06.573490 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e45b229c-4bb3-4736-a126-af6c6484f54c-tls-certs" pod "e45b229c-4bb3-4736-a126-af6c6484f54c" (UID: "e45b229c-4bb3-4736-a126-af6c6484f54c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:56:06.573602 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:56:06.573487 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e45b229c-4bb3-4736-a126-af6c6484f54c-dshm" pod "e45b229c-4bb3-4736-a126-af6c6484f54c" (UID: "e45b229c-4bb3-4736-a126-af6c6484f54c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:56:06.573602 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:56:06.573520 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e45b229c-4bb3-4736-a126-af6c6484f54c-kube-api-access-z42r2" pod "e45b229c-4bb3-4736-a126-af6c6484f54c" (UID: "e45b229c-4bb3-4736-a126-af6c6484f54c"). InnerVolumeSpecName "kube-api-access-z42r2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:56:06.672708 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:56:06.672674 5437 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e45b229c-4bb3-4736-a126-af6c6484f54c-dshm\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:56:06.672708 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:56:06.672701 5437 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e45b229c-4bb3-4736-a126-af6c6484f54c-tls-certs\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:56:06.672708 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:56:06.672712 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z42r2\" (UniqueName: \"kubernetes.io/projected/e45b229c-4bb3-4736-a126-af6c6484f54c-kube-api-access-z42r2\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:56:06.672948 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:56:06.672721 5437 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e45b229c-4bb3-4736-a126-af6c6484f54c-model-cache\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:56:06.984438 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:56:06.984405 5437 generic.go:356] "Generic (PLEG): container finished" podID="e45b229c-4bb3-4736-a126-af6c6484f54c" containerID="1139327ab10b4ca90dc4a09df976bf06b4fd2c4d7327033eaf7fee7b0f5d6367" exitCode=0 Nov 03 02:56:06.984627 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:56:06.984453 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-61825293-kserve-55f875484f-tndd6" Nov 03 02:56:06.984627 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:56:06.984502 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-61825293-kserve-55f875484f-tndd6" event={"ID":"e45b229c-4bb3-4736-a126-af6c6484f54c","Type":"ContainerDied","Data":"1139327ab10b4ca90dc4a09df976bf06b4fd2c4d7327033eaf7fee7b0f5d6367"} Nov 03 02:56:06.984627 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:56:06.984549 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-61825293-kserve-55f875484f-tndd6" event={"ID":"e45b229c-4bb3-4736-a126-af6c6484f54c","Type":"ContainerDied","Data":"daae2e337ebe9f59f5fcfba611fd9e30afa17f66a28116d32bbb3edcec677e86"} Nov 03 02:56:06.984627 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:56:06.984570 5437 scope.go:122] "RemoveContainer" containerID="1139327ab10b4ca90dc4a09df976bf06b4fd2c4d7327033eaf7fee7b0f5d6367" Nov 03 02:56:06.994564 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:56:06.994542 5437 scope.go:122] "RemoveContainer" containerID="1139327ab10b4ca90dc4a09df976bf06b4fd2c4d7327033eaf7fee7b0f5d6367" Nov 03 02:56:06.994852 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:56:06.994833 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1139327ab10b4ca90dc4a09df976bf06b4fd2c4d7327033eaf7fee7b0f5d6367\": container with ID starting with 1139327ab10b4ca90dc4a09df976bf06b4fd2c4d7327033eaf7fee7b0f5d6367 not found: ID does not exist" containerID="1139327ab10b4ca90dc4a09df976bf06b4fd2c4d7327033eaf7fee7b0f5d6367" Nov 03 02:56:06.994907 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:56:06.994861 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1139327ab10b4ca90dc4a09df976bf06b4fd2c4d7327033eaf7fee7b0f5d6367"} err="failed to get container status \"1139327ab10b4ca90dc4a09df976bf06b4fd2c4d7327033eaf7fee7b0f5d6367\": rpc error: code = NotFound desc = could not find container \"1139327ab10b4ca90dc4a09df976bf06b4fd2c4d7327033eaf7fee7b0f5d6367\": container with ID starting with 1139327ab10b4ca90dc4a09df976bf06b4fd2c4d7327033eaf7fee7b0f5d6367 not found: ID does not exist" Nov 03 02:56:07.155173 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:56:07.155136 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-61825293-kserve-55f875484f-tndd6"] Nov 03 02:56:07.159304 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:56:07.159280 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-61825293-kserve-55f875484f-tndd6"] Nov 03 02:56:08.087583 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:56:08.087549 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e45b229c-4bb3-4736-a126-af6c6484f54c" path="/var/lib/kubelet/pods/e45b229c-4bb3-4736-a126-af6c6484f54c/volumes" Nov 03 02:56:09.164946 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:56:09.164901 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:56:14.035355 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:56:14.035318 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 02:56:14.124972 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:56:14.124936 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:56:17.532664 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:56:17.532532 5437 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-dbf96 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 02:56:17.533327 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:56:17.533292 5437 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dbf96" podUID="fbdf72a4-6fc0-4511-b73e-ceabc4a76c2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 02:56:37.481995 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:56:37.481953 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-556d89986-6l4km"] Nov 03 02:56:37.482539 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:56:37.482359 5437 cpu_manager.go:414] "containerMap: removing container" podUID="e45b229c-4bb3-4736-a126-af6c6484f54c" containerName="main" Nov 03 02:56:37.482539 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:56:37.482385 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e45b229c-4bb3-4736-a126-af6c6484f54c" containerName="main" Nov 03 02:56:37.482539 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:56:37.482424 5437 cpu_manager.go:414] "containerMap: removing container" podUID="4dc38652-9cb8-4b45-bbc2-6e074a8554ba" containerName="main" Nov 03 02:56:37.482539 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:56:37.482429 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4dc38652-9cb8-4b45-bbc2-6e074a8554ba" containerName="main" Nov 03 02:56:37.482539 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:56:37.482489 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="4dc38652-9cb8-4b45-bbc2-6e074a8554ba" containerName="main" Nov 03 02:56:37.482539 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:56:37.482499 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="e45b229c-4bb3-4736-a126-af6c6484f54c" containerName="main" Nov 03 02:56:37.486758 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:56:37.486739 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-556d89986-6l4km" Nov 03 02:56:37.540854 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:56:37.540799 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-556d89986-6l4km"] Nov 03 02:56:37.608475 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:56:37.608438 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01b94c18-7aff-45af-ad47-8caa7202f5ec-plugins-config-volume\") pod \"payload-pre-processing-556d89986-6l4km\" (UID: \"01b94c18-7aff-45af-ad47-8caa7202f5ec\") " pod="openshift-ingress/payload-pre-processing-556d89986-6l4km" Nov 03 02:56:37.608670 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:56:37.608482 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7h4d\" (UniqueName: \"kubernetes.io/projected/01b94c18-7aff-45af-ad47-8caa7202f5ec-kube-api-access-d7h4d\") pod \"payload-pre-processing-556d89986-6l4km\" (UID: \"01b94c18-7aff-45af-ad47-8caa7202f5ec\") " pod="openshift-ingress/payload-pre-processing-556d89986-6l4km" Nov 03 02:56:37.709854 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:56:37.709816 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01b94c18-7aff-45af-ad47-8caa7202f5ec-plugins-config-volume\") pod \"payload-pre-processing-556d89986-6l4km\" (UID: \"01b94c18-7aff-45af-ad47-8caa7202f5ec\") " pod="openshift-ingress/payload-pre-processing-556d89986-6l4km" Nov 03 02:56:37.710049 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:56:37.709862 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7h4d\" (UniqueName: \"kubernetes.io/projected/01b94c18-7aff-45af-ad47-8caa7202f5ec-kube-api-access-d7h4d\") pod \"payload-pre-processing-556d89986-6l4km\" (UID: \"01b94c18-7aff-45af-ad47-8caa7202f5ec\") " pod="openshift-ingress/payload-pre-processing-556d89986-6l4km" Nov 03 02:56:37.710613 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:56:37.710590 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01b94c18-7aff-45af-ad47-8caa7202f5ec-plugins-config-volume\") pod \"payload-pre-processing-556d89986-6l4km\" (UID: \"01b94c18-7aff-45af-ad47-8caa7202f5ec\") " pod="openshift-ingress/payload-pre-processing-556d89986-6l4km" Nov 03 02:56:37.772829 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:56:37.772749 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7h4d\" (UniqueName: \"kubernetes.io/projected/01b94c18-7aff-45af-ad47-8caa7202f5ec-kube-api-access-d7h4d\") pod \"payload-pre-processing-556d89986-6l4km\" (UID: \"01b94c18-7aff-45af-ad47-8caa7202f5ec\") " pod="openshift-ingress/payload-pre-processing-556d89986-6l4km" Nov 03 02:56:37.799695 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:56:37.799652 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-556d89986-6l4km" Nov 03 02:56:38.169517 ip-10-0-136-16 kubenswrapper[5437]: W1103 02:56:38.169479 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01b94c18_7aff_45af_ad47_8caa7202f5ec.slice/crio-1ff124b3743fdce3c176dc4474fee11907d6d16d3ceff7fc660a4f1b48e64567 WatchSource:0}: Error finding container 1ff124b3743fdce3c176dc4474fee11907d6d16d3ceff7fc660a4f1b48e64567: Status 404 returned error can't find the container with id 1ff124b3743fdce3c176dc4474fee11907d6d16d3ceff7fc660a4f1b48e64567 Nov 03 02:56:38.171299 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:56:38.171259 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-556d89986-6l4km"] Nov 03 02:56:39.152757 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:56:39.152720 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-556d89986-6l4km" event={"ID":"01b94c18-7aff-45af-ad47-8caa7202f5ec","Type":"ContainerStarted","Data":"dd50dd2638d594d0230e4c35a673e6be4392582e727c3e5ca97654c7eeee3241"} Nov 03 02:56:39.153175 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:56:39.152781 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-556d89986-6l4km" event={"ID":"01b94c18-7aff-45af-ad47-8caa7202f5ec","Type":"ContainerStarted","Data":"1ff124b3743fdce3c176dc4474fee11907d6d16d3ceff7fc660a4f1b48e64567"} Nov 03 02:56:39.153175 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:56:39.152811 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-556d89986-6l4km" Nov 03 02:56:39.288184 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:56:39.288134 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-556d89986-6l4km" podStartSLOduration=2.288120499 podStartE2EDuration="2.288120499s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:56:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:56:39.287471006 +0000 UTC m=+5881.316953301" watchObservedRunningTime="2026-11-03 02:56:39.288120499 +0000 UTC m=+5881.317602816" Nov 03 02:56:44.034709 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:56:44.034664 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 02:56:50.160827 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:56:50.160791 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-556d89986-6l4km" Nov 03 02:56:50.373212 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:56:50.373173 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-jltq6"] Nov 03 02:56:55.373942 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:56:55.373879 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-7b98bb678c-jltq6" podUID="1748acc6-ac8d-4a30-ba58-70ade1bcbd0f" containerName="payload-pre-processing" containerID="cri-o://5df2b7978496690b78ef55df0bb656324505478d5c9f332d3ed784d87d336e15" gracePeriod=25 Nov 03 02:56:55.627982 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:56:55.627917 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-jltq6" Nov 03 02:56:55.685171 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:56:55.685133 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1748acc6-ac8d-4a30-ba58-70ade1bcbd0f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1748acc6-ac8d-4a30-ba58-70ade1bcbd0f-plugins-config-volume\") pod \"1748acc6-ac8d-4a30-ba58-70ade1bcbd0f\" (UID: \"1748acc6-ac8d-4a30-ba58-70ade1bcbd0f\") " Nov 03 02:56:55.685342 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:56:55.685207 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1748acc6-ac8d-4a30-ba58-70ade1bcbd0f-kube-api-access-sdbx2\" (UniqueName: \"kubernetes.io/projected/1748acc6-ac8d-4a30-ba58-70ade1bcbd0f-kube-api-access-sdbx2\") pod \"1748acc6-ac8d-4a30-ba58-70ade1bcbd0f\" (UID: \"1748acc6-ac8d-4a30-ba58-70ade1bcbd0f\") " Nov 03 02:56:55.685585 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:56:55.685560 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1748acc6-ac8d-4a30-ba58-70ade1bcbd0f-plugins-config-volume" pod "1748acc6-ac8d-4a30-ba58-70ade1bcbd0f" (UID: "1748acc6-ac8d-4a30-ba58-70ade1bcbd0f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:56:55.687351 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:56:55.687327 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1748acc6-ac8d-4a30-ba58-70ade1bcbd0f-kube-api-access-sdbx2" pod "1748acc6-ac8d-4a30-ba58-70ade1bcbd0f" (UID: "1748acc6-ac8d-4a30-ba58-70ade1bcbd0f"). InnerVolumeSpecName "kube-api-access-sdbx2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:56:55.786608 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:56:55.786578 5437 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1748acc6-ac8d-4a30-ba58-70ade1bcbd0f-plugins-config-volume\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:56:55.786608 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:56:55.786605 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdbx2\" (UniqueName: \"kubernetes.io/projected/1748acc6-ac8d-4a30-ba58-70ade1bcbd0f-kube-api-access-sdbx2\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:56:56.230219 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:56:56.230176 5437 generic.go:356] "Generic (PLEG): container finished" podID="1748acc6-ac8d-4a30-ba58-70ade1bcbd0f" containerID="5df2b7978496690b78ef55df0bb656324505478d5c9f332d3ed784d87d336e15" exitCode=0 Nov 03 02:56:56.230418 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:56:56.230242 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-jltq6" Nov 03 02:56:56.230418 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:56:56.230262 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-jltq6" event={"ID":"1748acc6-ac8d-4a30-ba58-70ade1bcbd0f","Type":"ContainerDied","Data":"5df2b7978496690b78ef55df0bb656324505478d5c9f332d3ed784d87d336e15"} Nov 03 02:56:56.230418 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:56:56.230318 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-jltq6" event={"ID":"1748acc6-ac8d-4a30-ba58-70ade1bcbd0f","Type":"ContainerDied","Data":"2f046892450de428d6cbbfdaebb6ff5b88103ba4ee4748f4b2da5102ec3c592b"} Nov 03 02:56:56.230418 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:56:56.230349 5437 scope.go:122] "RemoveContainer" containerID="5df2b7978496690b78ef55df0bb656324505478d5c9f332d3ed784d87d336e15" Nov 03 02:56:56.238879 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:56:56.238860 5437 scope.go:122] "RemoveContainer" containerID="5df2b7978496690b78ef55df0bb656324505478d5c9f332d3ed784d87d336e15" Nov 03 02:56:56.239140 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:56:56.239118 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5df2b7978496690b78ef55df0bb656324505478d5c9f332d3ed784d87d336e15\": container with ID starting with 5df2b7978496690b78ef55df0bb656324505478d5c9f332d3ed784d87d336e15 not found: ID does not exist" containerID="5df2b7978496690b78ef55df0bb656324505478d5c9f332d3ed784d87d336e15" Nov 03 02:56:56.239201 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:56:56.239148 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5df2b7978496690b78ef55df0bb656324505478d5c9f332d3ed784d87d336e15"} err="failed to get container status \"5df2b7978496690b78ef55df0bb656324505478d5c9f332d3ed784d87d336e15\": rpc error: code = NotFound desc = could not find container \"5df2b7978496690b78ef55df0bb656324505478d5c9f332d3ed784d87d336e15\": container with ID starting with 5df2b7978496690b78ef55df0bb656324505478d5c9f332d3ed784d87d336e15 not found: ID does not exist" Nov 03 02:56:56.402405 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:56:56.402320 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-jltq6"] Nov 03 02:56:56.453887 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:56:56.453846 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-jltq6"] Nov 03 02:56:58.087675 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:56:58.087642 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1748acc6-ac8d-4a30-ba58-70ade1bcbd0f" path="/var/lib/kubelet/pods/1748acc6-ac8d-4a30-ba58-70ade1bcbd0f/volumes" Nov 03 02:57:04.863731 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:57:04.863693 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-ghxht"] Nov 03 02:57:07.020215 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:57:07.020168 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-7987b45974-f58pp"] Nov 03 02:57:07.020678 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:57:07.020637 5437 cpu_manager.go:414] "containerMap: removing container" podUID="1748acc6-ac8d-4a30-ba58-70ade1bcbd0f" containerName="payload-pre-processing" Nov 03 02:57:07.020678 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:57:07.020650 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1748acc6-ac8d-4a30-ba58-70ade1bcbd0f" containerName="payload-pre-processing" Nov 03 02:57:07.020770 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:57:07.020725 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="1748acc6-ac8d-4a30-ba58-70ade1bcbd0f" containerName="payload-pre-processing" Nov 03 02:57:07.025063 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:57:07.025044 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-7987b45974-f58pp" Nov 03 02:57:07.079607 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:57:07.079572 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-7987b45974-f58pp"] Nov 03 02:57:07.105886 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:57:07.105846 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9faced96-b336-439d-824a-2dd51fbf8e00-plugins-config-volume\") pod \"payload-processing-7987b45974-f58pp\" (UID: \"9faced96-b336-439d-824a-2dd51fbf8e00\") " pod="openshift-ingress/payload-processing-7987b45974-f58pp" Nov 03 02:57:07.106084 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:57:07.106004 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxxz4\" (UniqueName: \"kubernetes.io/projected/9faced96-b336-439d-824a-2dd51fbf8e00-kube-api-access-hxxz4\") pod \"payload-processing-7987b45974-f58pp\" (UID: \"9faced96-b336-439d-824a-2dd51fbf8e00\") " pod="openshift-ingress/payload-processing-7987b45974-f58pp" Nov 03 02:57:07.206925 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:57:07.206884 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxxz4\" (UniqueName: \"kubernetes.io/projected/9faced96-b336-439d-824a-2dd51fbf8e00-kube-api-access-hxxz4\") pod \"payload-processing-7987b45974-f58pp\" (UID: \"9faced96-b336-439d-824a-2dd51fbf8e00\") " pod="openshift-ingress/payload-processing-7987b45974-f58pp" Nov 03 02:57:07.207114 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:57:07.206979 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9faced96-b336-439d-824a-2dd51fbf8e00-plugins-config-volume\") pod \"payload-processing-7987b45974-f58pp\" (UID: \"9faced96-b336-439d-824a-2dd51fbf8e00\") " pod="openshift-ingress/payload-processing-7987b45974-f58pp" Nov 03 02:57:07.207700 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:57:07.207678 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9faced96-b336-439d-824a-2dd51fbf8e00-plugins-config-volume\") pod \"payload-processing-7987b45974-f58pp\" (UID: \"9faced96-b336-439d-824a-2dd51fbf8e00\") " pod="openshift-ingress/payload-processing-7987b45974-f58pp" Nov 03 02:57:07.272132 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:57:07.272057 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxxz4\" (UniqueName: \"kubernetes.io/projected/9faced96-b336-439d-824a-2dd51fbf8e00-kube-api-access-hxxz4\") pod \"payload-processing-7987b45974-f58pp\" (UID: \"9faced96-b336-439d-824a-2dd51fbf8e00\") " pod="openshift-ingress/payload-processing-7987b45974-f58pp" Nov 03 02:57:07.338466 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:57:07.338430 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-7987b45974-f58pp" Nov 03 02:57:07.712023 ip-10-0-136-16 kubenswrapper[5437]: W1103 02:57:07.711990 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9faced96_b336_439d_824a_2dd51fbf8e00.slice/crio-6afe1531ea0865ae3d26bdeb0ab400d2ea0c0bf51a72e646db573dfb8f329ab4 WatchSource:0}: Error finding container 6afe1531ea0865ae3d26bdeb0ab400d2ea0c0bf51a72e646db573dfb8f329ab4: Status 404 returned error can't find the container with id 6afe1531ea0865ae3d26bdeb0ab400d2ea0c0bf51a72e646db573dfb8f329ab4 Nov 03 02:57:07.714455 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:57:07.714417 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-7987b45974-f58pp"] Nov 03 02:57:08.289877 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:57:08.289841 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-7987b45974-f58pp" event={"ID":"9faced96-b336-439d-824a-2dd51fbf8e00","Type":"ContainerStarted","Data":"2e908841c565bbab85a8b179803ad70a8af90faf6a7a9fb4c357aa2eaa604310"} Nov 03 02:57:08.290448 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:57:08.289895 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-7987b45974-f58pp" event={"ID":"9faced96-b336-439d-824a-2dd51fbf8e00","Type":"ContainerStarted","Data":"6afe1531ea0865ae3d26bdeb0ab400d2ea0c0bf51a72e646db573dfb8f329ab4"} Nov 03 02:57:08.290448 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:57:08.289940 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-7987b45974-f58pp" Nov 03 02:57:08.416406 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:57:08.412646 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-7987b45974-f58pp" podStartSLOduration=2.412623292 podStartE2EDuration="2.412623292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:57:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:57:08.409452633 +0000 UTC m=+5910.438934929" watchObservedRunningTime="2026-11-03 02:57:08.412623292 +0000 UTC m=+5910.442105589" Nov 03 02:57:09.519943 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:57:09.519909 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:57:09.864999 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:57:09.864952 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-55fbbbf9dc-ghxht" podUID="c4e1a934-d917-417b-aea3-d432d2cff990" containerName="payload-processing" containerID="cri-o://e9c6e7e5b1c0fde561e930dac94e21c3d03ffda78868acd88f239c428ffa0e47" gracePeriod=25 Nov 03 02:57:10.112485 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:57:10.112459 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-ghxht" Nov 03 02:57:10.236864 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:57:10.236773 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4e1a934-d917-417b-aea3-d432d2cff990-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4e1a934-d917-417b-aea3-d432d2cff990-plugins-config-volume\") pod \"c4e1a934-d917-417b-aea3-d432d2cff990\" (UID: \"c4e1a934-d917-417b-aea3-d432d2cff990\") " Nov 03 02:57:10.236864 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:57:10.236848 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4e1a934-d917-417b-aea3-d432d2cff990-kube-api-access-9cm99\" (UniqueName: \"kubernetes.io/projected/c4e1a934-d917-417b-aea3-d432d2cff990-kube-api-access-9cm99\") pod \"c4e1a934-d917-417b-aea3-d432d2cff990\" (UID: \"c4e1a934-d917-417b-aea3-d432d2cff990\") " Nov 03 02:57:10.237214 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:57:10.237192 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4e1a934-d917-417b-aea3-d432d2cff990-plugins-config-volume" pod "c4e1a934-d917-417b-aea3-d432d2cff990" (UID: "c4e1a934-d917-417b-aea3-d432d2cff990"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:57:10.239665 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:57:10.239633 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4e1a934-d917-417b-aea3-d432d2cff990-kube-api-access-9cm99" pod "c4e1a934-d917-417b-aea3-d432d2cff990" (UID: "c4e1a934-d917-417b-aea3-d432d2cff990"). InnerVolumeSpecName "kube-api-access-9cm99". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:57:10.301575 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:57:10.301542 5437 generic.go:356] "Generic (PLEG): container finished" podID="c4e1a934-d917-417b-aea3-d432d2cff990" containerID="e9c6e7e5b1c0fde561e930dac94e21c3d03ffda78868acd88f239c428ffa0e47" exitCode=0 Nov 03 02:57:10.301757 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:57:10.301595 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-ghxht" Nov 03 02:57:10.301757 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:57:10.301629 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-ghxht" event={"ID":"c4e1a934-d917-417b-aea3-d432d2cff990","Type":"ContainerDied","Data":"e9c6e7e5b1c0fde561e930dac94e21c3d03ffda78868acd88f239c428ffa0e47"} Nov 03 02:57:10.301757 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:57:10.301691 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-ghxht" event={"ID":"c4e1a934-d917-417b-aea3-d432d2cff990","Type":"ContainerDied","Data":"6965aaf93fd6b13fa4c3cda4b0e13b975c5aa6bae07227a4b21d01c4ddc628aa"} Nov 03 02:57:10.301757 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:57:10.301720 5437 scope.go:122] "RemoveContainer" containerID="e9c6e7e5b1c0fde561e930dac94e21c3d03ffda78868acd88f239c428ffa0e47" Nov 03 02:57:10.310698 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:57:10.310676 5437 scope.go:122] "RemoveContainer" containerID="e9c6e7e5b1c0fde561e930dac94e21c3d03ffda78868acd88f239c428ffa0e47" Nov 03 02:57:10.310988 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:57:10.310966 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9c6e7e5b1c0fde561e930dac94e21c3d03ffda78868acd88f239c428ffa0e47\": container with ID starting with e9c6e7e5b1c0fde561e930dac94e21c3d03ffda78868acd88f239c428ffa0e47 not found: ID does not exist" containerID="e9c6e7e5b1c0fde561e930dac94e21c3d03ffda78868acd88f239c428ffa0e47" Nov 03 02:57:10.311073 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:57:10.310996 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9c6e7e5b1c0fde561e930dac94e21c3d03ffda78868acd88f239c428ffa0e47"} err="failed to get container status \"e9c6e7e5b1c0fde561e930dac94e21c3d03ffda78868acd88f239c428ffa0e47\": rpc error: code = NotFound desc = could not find container \"e9c6e7e5b1c0fde561e930dac94e21c3d03ffda78868acd88f239c428ffa0e47\": container with ID starting with e9c6e7e5b1c0fde561e930dac94e21c3d03ffda78868acd88f239c428ffa0e47 not found: ID does not exist" Nov 03 02:57:10.338595 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:57:10.338565 5437 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4e1a934-d917-417b-aea3-d432d2cff990-plugins-config-volume\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:57:10.338595 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:57:10.338593 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cm99\" (UniqueName: \"kubernetes.io/projected/c4e1a934-d917-417b-aea3-d432d2cff990-kube-api-access-9cm99\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:57:10.471282 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:57:10.471245 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-ghxht"] Nov 03 02:57:10.475677 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:57:10.475652 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-ghxht"] Nov 03 02:57:12.087161 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:57:12.087128 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4e1a934-d917-417b-aea3-d432d2cff990" path="/var/lib/kubelet/pods/c4e1a934-d917-417b-aea3-d432d2cff990/volumes" Nov 03 02:57:14.035860 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:57:14.035823 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 02:57:14.478837 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:57:14.478791 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:57:19.299071 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:57:19.299042 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-7987b45974-f58pp" Nov 03 02:57:19.952608 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:57:19.952570 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-556d89986-6l4km"] Nov 03 02:57:22.555035 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:57:22.554999 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:57:22.977335 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:57:22.977248 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:57:24.953624 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:57:24.953556 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-556d89986-6l4km" podUID="01b94c18-7aff-45af-ad47-8caa7202f5ec" containerName="payload-pre-processing" containerID="cri-o://dd50dd2638d594d0230e4c35a673e6be4392582e727c3e5ca97654c7eeee3241" gracePeriod=25 Nov 03 02:57:25.203755 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:57:25.203692 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-556d89986-6l4km" Nov 03 02:57:25.297418 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:57:25.297381 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01b94c18-7aff-45af-ad47-8caa7202f5ec-kube-api-access-d7h4d\" (UniqueName: \"kubernetes.io/projected/01b94c18-7aff-45af-ad47-8caa7202f5ec-kube-api-access-d7h4d\") pod \"01b94c18-7aff-45af-ad47-8caa7202f5ec\" (UID: \"01b94c18-7aff-45af-ad47-8caa7202f5ec\") " Nov 03 02:57:25.297620 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:57:25.297520 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01b94c18-7aff-45af-ad47-8caa7202f5ec-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01b94c18-7aff-45af-ad47-8caa7202f5ec-plugins-config-volume\") pod \"01b94c18-7aff-45af-ad47-8caa7202f5ec\" (UID: \"01b94c18-7aff-45af-ad47-8caa7202f5ec\") " Nov 03 02:57:25.297937 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:57:25.297912 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01b94c18-7aff-45af-ad47-8caa7202f5ec-plugins-config-volume" pod "01b94c18-7aff-45af-ad47-8caa7202f5ec" (UID: "01b94c18-7aff-45af-ad47-8caa7202f5ec"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:57:25.299568 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:57:25.299543 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01b94c18-7aff-45af-ad47-8caa7202f5ec-kube-api-access-d7h4d" pod "01b94c18-7aff-45af-ad47-8caa7202f5ec" (UID: "01b94c18-7aff-45af-ad47-8caa7202f5ec"). InnerVolumeSpecName "kube-api-access-d7h4d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:57:25.378002 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:57:25.377968 5437 generic.go:356] "Generic (PLEG): container finished" podID="01b94c18-7aff-45af-ad47-8caa7202f5ec" containerID="dd50dd2638d594d0230e4c35a673e6be4392582e727c3e5ca97654c7eeee3241" exitCode=0 Nov 03 02:57:25.378168 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:57:25.378013 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-556d89986-6l4km" Nov 03 02:57:25.378168 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:57:25.378052 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-556d89986-6l4km" event={"ID":"01b94c18-7aff-45af-ad47-8caa7202f5ec","Type":"ContainerDied","Data":"dd50dd2638d594d0230e4c35a673e6be4392582e727c3e5ca97654c7eeee3241"} Nov 03 02:57:25.378168 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:57:25.378134 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-556d89986-6l4km" event={"ID":"01b94c18-7aff-45af-ad47-8caa7202f5ec","Type":"ContainerDied","Data":"1ff124b3743fdce3c176dc4474fee11907d6d16d3ceff7fc660a4f1b48e64567"} Nov 03 02:57:25.378168 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:57:25.378163 5437 scope.go:122] "RemoveContainer" containerID="dd50dd2638d594d0230e4c35a673e6be4392582e727c3e5ca97654c7eeee3241" Nov 03 02:57:25.388107 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:57:25.388081 5437 scope.go:122] "RemoveContainer" containerID="dd50dd2638d594d0230e4c35a673e6be4392582e727c3e5ca97654c7eeee3241" Nov 03 02:57:25.388479 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:57:25.388450 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd50dd2638d594d0230e4c35a673e6be4392582e727c3e5ca97654c7eeee3241\": container with ID starting with dd50dd2638d594d0230e4c35a673e6be4392582e727c3e5ca97654c7eeee3241 not found: ID does not exist" containerID="dd50dd2638d594d0230e4c35a673e6be4392582e727c3e5ca97654c7eeee3241" Nov 03 02:57:25.388580 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:57:25.388487 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd50dd2638d594d0230e4c35a673e6be4392582e727c3e5ca97654c7eeee3241"} err="failed to get container status \"dd50dd2638d594d0230e4c35a673e6be4392582e727c3e5ca97654c7eeee3241\": rpc error: code = NotFound desc = could not find container \"dd50dd2638d594d0230e4c35a673e6be4392582e727c3e5ca97654c7eeee3241\": container with ID starting with dd50dd2638d594d0230e4c35a673e6be4392582e727c3e5ca97654c7eeee3241 not found: ID does not exist" Nov 03 02:57:25.399743 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:57:25.399714 5437 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01b94c18-7aff-45af-ad47-8caa7202f5ec-plugins-config-volume\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:57:25.399743 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:57:25.399743 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7h4d\" (UniqueName: \"kubernetes.io/projected/01b94c18-7aff-45af-ad47-8caa7202f5ec-kube-api-access-d7h4d\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:57:25.555067 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:57:25.555021 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-556d89986-6l4km"] Nov 03 02:57:25.560047 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:57:25.560022 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-556d89986-6l4km"] Nov 03 02:57:25.660315 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:57:25.660278 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:57:26.087470 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:57:26.087435 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01b94c18-7aff-45af-ad47-8caa7202f5ec" path="/var/lib/kubelet/pods/01b94c18-7aff-45af-ad47-8caa7202f5ec/volumes" Nov 03 02:57:26.109776 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:57:26.109746 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-fs6ld"] Nov 03 02:57:26.110155 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:57:26.110142 5437 cpu_manager.go:414] "containerMap: removing container" podUID="c4e1a934-d917-417b-aea3-d432d2cff990" containerName="payload-processing" Nov 03 02:57:26.110200 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:57:26.110157 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4e1a934-d917-417b-aea3-d432d2cff990" containerName="payload-processing" Nov 03 02:57:26.110200 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:57:26.110170 5437 cpu_manager.go:414] "containerMap: removing container" podUID="01b94c18-7aff-45af-ad47-8caa7202f5ec" containerName="payload-pre-processing" Nov 03 02:57:26.110200 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:57:26.110175 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01b94c18-7aff-45af-ad47-8caa7202f5ec" containerName="payload-pre-processing" Nov 03 02:57:26.110298 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:57:26.110252 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4e1a934-d917-417b-aea3-d432d2cff990" containerName="payload-processing" Nov 03 02:57:26.110298 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:57:26.110260 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="01b94c18-7aff-45af-ad47-8caa7202f5ec" containerName="payload-pre-processing" Nov 03 02:57:26.113182 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:57:26.113161 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-fs6ld" Nov 03 02:57:26.168899 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:57:26.168866 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-fs6ld"] Nov 03 02:57:26.173793 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:57:26.173770 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Nov 03 02:57:26.207450 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:57:26.207416 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tlcz\" (UniqueName: \"kubernetes.io/projected/d0a0c1a3-5e58-465a-9507-87c0138314be-kube-api-access-6tlcz\") pod \"payload-processing-external-model-5bcd7997-fs6ld\" (UID: \"d0a0c1a3-5e58-465a-9507-87c0138314be\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-fs6ld" Nov 03 02:57:26.207644 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:57:26.207500 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0a0c1a3-5e58-465a-9507-87c0138314be-plugins-config-volume\") pod \"payload-processing-external-model-5bcd7997-fs6ld\" (UID: \"d0a0c1a3-5e58-465a-9507-87c0138314be\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-fs6ld" Nov 03 02:57:26.207644 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:57:26.207623 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d0a0c1a3-5e58-465a-9507-87c0138314be-routing-overlay-volume\") pod \"payload-processing-external-model-5bcd7997-fs6ld\" (UID: \"d0a0c1a3-5e58-465a-9507-87c0138314be\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-fs6ld" Nov 03 02:57:26.207746 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:57:26.207658 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d0a0c1a3-5e58-465a-9507-87c0138314be-provider-credentials\") pod \"payload-processing-external-model-5bcd7997-fs6ld\" (UID: \"d0a0c1a3-5e58-465a-9507-87c0138314be\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-fs6ld" Nov 03 02:57:26.218529 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:57:26.218498 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Nov 03 02:57:26.218673 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:57:26.218585 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Nov 03 02:57:26.218719 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:57:26.218707 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"e2e-rotation-external-model-api-key\"" Nov 03 02:57:26.218795 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:57:26.218780 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Nov 03 02:57:26.218843 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:57:26.218786 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-zfcqw\"" Nov 03 02:57:26.308846 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:57:26.308809 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d0a0c1a3-5e58-465a-9507-87c0138314be-routing-overlay-volume\") pod \"payload-processing-external-model-5bcd7997-fs6ld\" (UID: \"d0a0c1a3-5e58-465a-9507-87c0138314be\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-fs6ld" Nov 03 02:57:26.309055 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:57:26.308856 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d0a0c1a3-5e58-465a-9507-87c0138314be-provider-credentials\") pod \"payload-processing-external-model-5bcd7997-fs6ld\" (UID: \"d0a0c1a3-5e58-465a-9507-87c0138314be\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-fs6ld" Nov 03 02:57:26.309055 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:57:26.308902 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tlcz\" (UniqueName: \"kubernetes.io/projected/d0a0c1a3-5e58-465a-9507-87c0138314be-kube-api-access-6tlcz\") pod \"payload-processing-external-model-5bcd7997-fs6ld\" (UID: \"d0a0c1a3-5e58-465a-9507-87c0138314be\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-fs6ld" Nov 03 02:57:26.309055 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:57:26.308986 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0a0c1a3-5e58-465a-9507-87c0138314be-plugins-config-volume\") pod \"payload-processing-external-model-5bcd7997-fs6ld\" (UID: \"d0a0c1a3-5e58-465a-9507-87c0138314be\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-fs6ld" Nov 03 02:57:26.309667 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:57:26.309640 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d0a0c1a3-5e58-465a-9507-87c0138314be-routing-overlay-volume\") pod \"payload-processing-external-model-5bcd7997-fs6ld\" (UID: \"d0a0c1a3-5e58-465a-9507-87c0138314be\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-fs6ld" Nov 03 02:57:26.309798 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:57:26.309763 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0a0c1a3-5e58-465a-9507-87c0138314be-plugins-config-volume\") pod \"payload-processing-external-model-5bcd7997-fs6ld\" (UID: \"d0a0c1a3-5e58-465a-9507-87c0138314be\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-fs6ld" Nov 03 02:57:26.311545 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:57:26.311523 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d0a0c1a3-5e58-465a-9507-87c0138314be-provider-credentials\") pod \"payload-processing-external-model-5bcd7997-fs6ld\" (UID: \"d0a0c1a3-5e58-465a-9507-87c0138314be\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-fs6ld" Nov 03 02:57:26.371737 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:57:26.371667 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tlcz\" (UniqueName: \"kubernetes.io/projected/d0a0c1a3-5e58-465a-9507-87c0138314be-kube-api-access-6tlcz\") pod \"payload-processing-external-model-5bcd7997-fs6ld\" (UID: \"d0a0c1a3-5e58-465a-9507-87c0138314be\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-fs6ld" Nov 03 02:57:26.426051 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:57:26.426012 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-fs6ld" Nov 03 02:57:26.816869 ip-10-0-136-16 kubenswrapper[5437]: W1103 02:57:26.816836 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0a0c1a3_5e58_465a_9507_87c0138314be.slice/crio-b7a8dc0b823ef0ec040254e883e69b15e8266cd0de24d145a0a01e05aed2a84f WatchSource:0}: Error finding container b7a8dc0b823ef0ec040254e883e69b15e8266cd0de24d145a0a01e05aed2a84f: Status 404 returned error can't find the container with id b7a8dc0b823ef0ec040254e883e69b15e8266cd0de24d145a0a01e05aed2a84f Nov 03 02:57:26.819763 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:57:26.819732 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-fs6ld"] Nov 03 02:57:27.390555 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:57:27.390517 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-fs6ld" event={"ID":"d0a0c1a3-5e58-465a-9507-87c0138314be","Type":"ContainerStarted","Data":"fcaf10ba2e24d2489575ddb5be882ad12b243c439cb795fa9032b55affb77ca2"} Nov 03 02:57:27.391040 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:57:27.390571 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-fs6ld" event={"ID":"d0a0c1a3-5e58-465a-9507-87c0138314be","Type":"ContainerStarted","Data":"b7a8dc0b823ef0ec040254e883e69b15e8266cd0de24d145a0a01e05aed2a84f"} Nov 03 02:57:27.391040 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:57:27.390606 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-5bcd7997-fs6ld" Nov 03 02:57:27.517552 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:57:27.517497 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-5bcd7997-fs6ld" podStartSLOduration=1.517478337 podStartE2EDuration="1.517478337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:57:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:57:27.516061662 +0000 UTC m=+5929.545543957" watchObservedRunningTime="2026-11-03 02:57:27.517478337 +0000 UTC m=+5929.546960632" Nov 03 02:57:30.656978 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:57:30.656944 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:57:38.401524 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:57:38.401492 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-5bcd7997-fs6ld" Nov 03 02:57:44.034898 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:57:44.034854 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 02:57:52.709356 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:57:52.709262 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:58:09.868784 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:58:09.868738 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:58:14.034999 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:58:14.034952 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 02:58:14.821394 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:58:14.821347 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:58:44.034340 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:58:44.034300 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 02:59:01.734785 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:59:01.734744 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-fs6ld"] Nov 03 02:59:01.948119 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:59:01.948079 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:59:03.969802 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:59:03.969749 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:59:06.738498 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:59:06.738444 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-5bcd7997-fs6ld" podUID="d0a0c1a3-5e58-465a-9507-87c0138314be" containerName="payload-processing" containerID="cri-o://fcaf10ba2e24d2489575ddb5be882ad12b243c439cb795fa9032b55affb77ca2" gracePeriod=25 Nov 03 02:59:07.119751 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:59:07.119718 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-fs6ld" Nov 03 02:59:07.302826 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:59:07.302797 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0a0c1a3-5e58-465a-9507-87c0138314be-kube-api-access-6tlcz\" (UniqueName: \"kubernetes.io/projected/d0a0c1a3-5e58-465a-9507-87c0138314be-kube-api-access-6tlcz\") pod \"d0a0c1a3-5e58-465a-9507-87c0138314be\" (UID: \"d0a0c1a3-5e58-465a-9507-87c0138314be\") " Nov 03 02:59:07.302999 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:59:07.302852 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0a0c1a3-5e58-465a-9507-87c0138314be-provider-credentials\" (UniqueName: \"kubernetes.io/projected/d0a0c1a3-5e58-465a-9507-87c0138314be-provider-credentials\") pod \"d0a0c1a3-5e58-465a-9507-87c0138314be\" (UID: \"d0a0c1a3-5e58-465a-9507-87c0138314be\") " Nov 03 02:59:07.302999 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:59:07.302903 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0a0c1a3-5e58-465a-9507-87c0138314be-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0a0c1a3-5e58-465a-9507-87c0138314be-plugins-config-volume\") pod \"d0a0c1a3-5e58-465a-9507-87c0138314be\" (UID: \"d0a0c1a3-5e58-465a-9507-87c0138314be\") " Nov 03 02:59:07.303120 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:59:07.303004 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0a0c1a3-5e58-465a-9507-87c0138314be-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d0a0c1a3-5e58-465a-9507-87c0138314be-routing-overlay-volume\") pod \"d0a0c1a3-5e58-465a-9507-87c0138314be\" (UID: \"d0a0c1a3-5e58-465a-9507-87c0138314be\") " Nov 03 02:59:07.303372 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:59:07.303337 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0a0c1a3-5e58-465a-9507-87c0138314be-routing-overlay-volume" pod "d0a0c1a3-5e58-465a-9507-87c0138314be" (UID: "d0a0c1a3-5e58-465a-9507-87c0138314be"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:59:07.303511 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:59:07.303354 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0a0c1a3-5e58-465a-9507-87c0138314be-plugins-config-volume" pod "d0a0c1a3-5e58-465a-9507-87c0138314be" (UID: "d0a0c1a3-5e58-465a-9507-87c0138314be"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:59:07.305051 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:59:07.305024 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0a0c1a3-5e58-465a-9507-87c0138314be-provider-credentials" pod "d0a0c1a3-5e58-465a-9507-87c0138314be" (UID: "d0a0c1a3-5e58-465a-9507-87c0138314be"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:59:07.305161 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:59:07.305072 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0a0c1a3-5e58-465a-9507-87c0138314be-kube-api-access-6tlcz" pod "d0a0c1a3-5e58-465a-9507-87c0138314be" (UID: "d0a0c1a3-5e58-465a-9507-87c0138314be"). InnerVolumeSpecName "kube-api-access-6tlcz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:59:07.403858 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:59:07.403823 5437 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d0a0c1a3-5e58-465a-9507-87c0138314be-provider-credentials\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:59:07.403858 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:59:07.403850 5437 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0a0c1a3-5e58-465a-9507-87c0138314be-plugins-config-volume\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:59:07.403858 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:59:07.403861 5437 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d0a0c1a3-5e58-465a-9507-87c0138314be-routing-overlay-volume\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:59:07.404098 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:59:07.403870 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6tlcz\" (UniqueName: \"kubernetes.io/projected/d0a0c1a3-5e58-465a-9507-87c0138314be-kube-api-access-6tlcz\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:59:07.900221 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:59:07.900185 5437 generic.go:356] "Generic (PLEG): container finished" podID="d0a0c1a3-5e58-465a-9507-87c0138314be" containerID="fcaf10ba2e24d2489575ddb5be882ad12b243c439cb795fa9032b55affb77ca2" exitCode=0 Nov 03 02:59:07.900673 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:59:07.900240 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-fs6ld" Nov 03 02:59:07.900673 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:59:07.900276 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-fs6ld" event={"ID":"d0a0c1a3-5e58-465a-9507-87c0138314be","Type":"ContainerDied","Data":"fcaf10ba2e24d2489575ddb5be882ad12b243c439cb795fa9032b55affb77ca2"} Nov 03 02:59:07.900673 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:59:07.900383 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-fs6ld" event={"ID":"d0a0c1a3-5e58-465a-9507-87c0138314be","Type":"ContainerDied","Data":"b7a8dc0b823ef0ec040254e883e69b15e8266cd0de24d145a0a01e05aed2a84f"} Nov 03 02:59:07.900673 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:59:07.900414 5437 scope.go:122] "RemoveContainer" containerID="fcaf10ba2e24d2489575ddb5be882ad12b243c439cb795fa9032b55affb77ca2" Nov 03 02:59:07.909626 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:59:07.909603 5437 scope.go:122] "RemoveContainer" containerID="fcaf10ba2e24d2489575ddb5be882ad12b243c439cb795fa9032b55affb77ca2" Nov 03 02:59:07.909924 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:59:07.909896 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcaf10ba2e24d2489575ddb5be882ad12b243c439cb795fa9032b55affb77ca2\": container with ID starting with fcaf10ba2e24d2489575ddb5be882ad12b243c439cb795fa9032b55affb77ca2 not found: ID does not exist" containerID="fcaf10ba2e24d2489575ddb5be882ad12b243c439cb795fa9032b55affb77ca2" Nov 03 02:59:07.910061 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:59:07.909931 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcaf10ba2e24d2489575ddb5be882ad12b243c439cb795fa9032b55affb77ca2"} err="failed to get container status \"fcaf10ba2e24d2489575ddb5be882ad12b243c439cb795fa9032b55affb77ca2\": rpc error: code = NotFound desc = could not find container \"fcaf10ba2e24d2489575ddb5be882ad12b243c439cb795fa9032b55affb77ca2\": container with ID starting with fcaf10ba2e24d2489575ddb5be882ad12b243c439cb795fa9032b55affb77ca2 not found: ID does not exist" Nov 03 02:59:08.069268 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:59:08.069231 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-fs6ld"] Nov 03 02:59:08.120760 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:59:08.120717 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-fs6ld"] Nov 03 02:59:10.087676 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:59:10.087638 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0a0c1a3-5e58-465a-9507-87c0138314be" path="/var/lib/kubelet/pods/d0a0c1a3-5e58-465a-9507-87c0138314be/volumes" Nov 03 02:59:10.206681 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:59:10.206646 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:59:14.034665 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:59:14.034631 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 02:59:15.150783 ip-10-0-136-16 kubenswrapper[5437]: I1103 02:59:15.150745 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 02:59:44.034130 ip-10-0-136-16 kubenswrapper[5437]: E1103 02:59:44.034088 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 03:00:10.620005 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:00:10.619954 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 03:00:14.033196 ip-10-0-136-16 kubenswrapper[5437]: E1103 03:00:14.033157 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 03:00:15.475908 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:00:15.475874 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 03:00:44.034484 ip-10-0-136-16 kubenswrapper[5437]: E1103 03:00:44.034445 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 03:00:50.336157 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:00:50.336063 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 03:00:50.546074 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:00:50.546032 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 03:00:57.141834 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:00:57.141785 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 03:01:10.934554 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:01:10.934507 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 03:01:14.034736 ip-10-0-136-16 kubenswrapper[5437]: E1103 03:01:14.034692 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 03:01:15.823386 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:01:15.823337 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 03:01:34.010111 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:01:34.010067 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 03:01:34.261613 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:01:34.261534 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 03:01:44.033783 ip-10-0-136-16 kubenswrapper[5437]: E1103 03:01:44.033744 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 03:02:11.302907 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:02:11.302864 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 03:02:11.508510 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:02:11.508470 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 03:02:14.034199 ip-10-0-136-16 kubenswrapper[5437]: E1103 03:02:14.034164 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 03:02:16.165993 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:02:16.165954 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 03:02:44.036628 ip-10-0-136-16 kubenswrapper[5437]: E1103 03:02:44.036580 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 03:02:46.547218 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:02:46.547182 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 03:03:03.164706 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:03:03.164674 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 03:03:11.635533 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:03:11.635498 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 03:03:14.033844 ip-10-0-136-16 kubenswrapper[5437]: E1103 03:03:14.033806 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 03:03:16.538388 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:03:16.537418 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 03:03:17.529392 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:03:17.523464 5437 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-dbf96 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 03:03:17.529392 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:03:17.523568 5437 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dbf96" podUID="fbdf72a4-6fc0-4511-b73e-ceabc4a76c2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 03:03:35.796186 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:03:35.796146 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 03:03:36.050819 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:03:36.050786 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 03:03:44.034127 ip-10-0-136-16 kubenswrapper[5437]: E1103 03:03:44.034089 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 03:04:09.280877 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:04:09.280798 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 03:04:11.599758 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:04:11.599720 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 03:04:11.995053 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:04:11.994960 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 03:04:14.034666 ip-10-0-136-16 kubenswrapper[5437]: E1103 03:04:14.034613 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 03:04:14.306720 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:04:14.306684 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 03:04:14.560742 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:04:14.560652 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 03:04:14.775866 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:04:14.775828 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 03:04:16.875125 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:04:16.875081 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 03:04:17.578982 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:04:17.578941 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 03:04:17.833916 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:04:17.833824 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 03:04:17.932625 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:04:17.932587 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-9fbc1b-55c4br4hj9"] Nov 03 03:04:17.933019 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:04:17.933004 5437 cpu_manager.go:414] "containerMap: removing container" podUID="d0a0c1a3-5e58-465a-9507-87c0138314be" containerName="payload-processing" Nov 03 03:04:17.933019 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:04:17.933017 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0a0c1a3-5e58-465a-9507-87c0138314be" containerName="payload-processing" Nov 03 03:04:17.933136 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:04:17.933124 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0a0c1a3-5e58-465a-9507-87c0138314be" containerName="payload-processing" Nov 03 03:04:17.936323 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:04:17.936301 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-9fbc1b-55c4br4hj9" Nov 03 03:04:17.990294 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:04:17.990251 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-9fbc1b-55c4br4hj9"] Nov 03 03:04:18.040403 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:04:18.040355 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-7cb3e802-dockercfg-8xvvd\"" Nov 03 03:04:18.040631 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:04:18.040431 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-9fbc1b\"" Nov 03 03:04:18.084978 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:04:18.084890 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-689bt\" (UniqueName: \"kubernetes.io/projected/fa041808-23c0-4527-aa7f-86e2bfe34e47-kube-api-access-689bt\") pod \"payload-pre-processing-e2e-worker-single-main-9fbc1b-55c4br4hj9\" (UID: \"fa041808-23c0-4527-aa7f-86e2bfe34e47\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-9fbc1b-55c4br4hj9" Nov 03 03:04:18.084978 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:04:18.084954 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa041808-23c0-4527-aa7f-86e2bfe34e47-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-9fbc1b-55c4br4hj9\" (UID: \"fa041808-23c0-4527-aa7f-86e2bfe34e47\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-9fbc1b-55c4br4hj9" Nov 03 03:04:18.093672 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:04:18.093638 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-9fbc1b-6df49d566v5qz7"] Nov 03 03:04:18.096975 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:04:18.096953 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-9fbc1b-6df49d566v5qz7" Nov 03 03:04:18.142282 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:04:18.142246 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-9fbc1b-6df49d566v5qz7"] Nov 03 03:04:18.185614 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:04:18.185566 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ngtt\" (UniqueName: \"kubernetes.io/projected/accac0db-76e1-45ae-b6b3-d637fd501261-kube-api-access-2ngtt\") pod \"payload-processing-e2e-worker-single-main-9fbc1b-6df49d566v5qz7\" (UID: \"accac0db-76e1-45ae-b6b3-d637fd501261\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-9fbc1b-6df49d566v5qz7" Nov 03 03:04:18.185614 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:04:18.185619 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa041808-23c0-4527-aa7f-86e2bfe34e47-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-9fbc1b-55c4br4hj9\" (UID: \"fa041808-23c0-4527-aa7f-86e2bfe34e47\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-9fbc1b-55c4br4hj9" Nov 03 03:04:18.185918 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:04:18.185879 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/accac0db-76e1-45ae-b6b3-d637fd501261-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-9fbc1b-6df49d566v5qz7\" (UID: \"accac0db-76e1-45ae-b6b3-d637fd501261\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-9fbc1b-6df49d566v5qz7" Nov 03 03:04:18.186007 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:04:18.185985 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-689bt\" (UniqueName: \"kubernetes.io/projected/fa041808-23c0-4527-aa7f-86e2bfe34e47-kube-api-access-689bt\") pod \"payload-pre-processing-e2e-worker-single-main-9fbc1b-55c4br4hj9\" (UID: \"fa041808-23c0-4527-aa7f-86e2bfe34e47\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-9fbc1b-55c4br4hj9" Nov 03 03:04:18.186465 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:04:18.186446 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa041808-23c0-4527-aa7f-86e2bfe34e47-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-9fbc1b-55c4br4hj9\" (UID: \"fa041808-23c0-4527-aa7f-86e2bfe34e47\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-9fbc1b-55c4br4hj9" Nov 03 03:04:18.249585 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:04:18.249545 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-689bt\" (UniqueName: \"kubernetes.io/projected/fa041808-23c0-4527-aa7f-86e2bfe34e47-kube-api-access-689bt\") pod \"payload-pre-processing-e2e-worker-single-main-9fbc1b-55c4br4hj9\" (UID: \"fa041808-23c0-4527-aa7f-86e2bfe34e47\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-9fbc1b-55c4br4hj9" Nov 03 03:04:18.287590 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:04:18.287543 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ngtt\" (UniqueName: \"kubernetes.io/projected/accac0db-76e1-45ae-b6b3-d637fd501261-kube-api-access-2ngtt\") pod \"payload-processing-e2e-worker-single-main-9fbc1b-6df49d566v5qz7\" (UID: \"accac0db-76e1-45ae-b6b3-d637fd501261\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-9fbc1b-6df49d566v5qz7" Nov 03 03:04:18.287779 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:04:18.287667 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/accac0db-76e1-45ae-b6b3-d637fd501261-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-9fbc1b-6df49d566v5qz7\" (UID: \"accac0db-76e1-45ae-b6b3-d637fd501261\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-9fbc1b-6df49d566v5qz7" Nov 03 03:04:18.288401 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:04:18.288352 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/accac0db-76e1-45ae-b6b3-d637fd501261-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-9fbc1b-6df49d566v5qz7\" (UID: \"accac0db-76e1-45ae-b6b3-d637fd501261\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-9fbc1b-6df49d566v5qz7" Nov 03 03:04:18.351152 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:04:18.351071 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ngtt\" (UniqueName: \"kubernetes.io/projected/accac0db-76e1-45ae-b6b3-d637fd501261-kube-api-access-2ngtt\") pod \"payload-processing-e2e-worker-single-main-9fbc1b-6df49d566v5qz7\" (UID: \"accac0db-76e1-45ae-b6b3-d637fd501261\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-9fbc1b-6df49d566v5qz7" Nov 03 03:04:18.409903 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:04:18.409866 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-9fbc1b-6df49d566v5qz7" Nov 03 03:04:18.549377 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:04:18.549323 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-9fbc1b-55c4br4hj9" Nov 03 03:04:18.780450 ip-10-0-136-16 kubenswrapper[5437]: W1103 03:04:18.780417 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaccac0db_76e1_45ae_b6b3_d637fd501261.slice/crio-7032106103c4eb0b4badf232f1d55b41abcdc7bf252e86f4d00bf9e09d4913b9 WatchSource:0}: Error finding container 7032106103c4eb0b4badf232f1d55b41abcdc7bf252e86f4d00bf9e09d4913b9: Status 404 returned error can't find the container with id 7032106103c4eb0b4badf232f1d55b41abcdc7bf252e86f4d00bf9e09d4913b9 Nov 03 03:04:18.782515 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:04:18.782467 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-9fbc1b-6df49d566v5qz7"] Nov 03 03:04:18.914620 ip-10-0-136-16 kubenswrapper[5437]: W1103 03:04:18.914591 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa041808_23c0_4527_aa7f_86e2bfe34e47.slice/crio-b41b4f73167d2dfad87a457abc957e707290e34158c96a30ffdb82e6a6888acc WatchSource:0}: Error finding container b41b4f73167d2dfad87a457abc957e707290e34158c96a30ffdb82e6a6888acc: Status 404 returned error can't find the container with id b41b4f73167d2dfad87a457abc957e707290e34158c96a30ffdb82e6a6888acc Nov 03 03:04:18.916937 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:04:18.916914 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-9fbc1b-55c4br4hj9"] Nov 03 03:04:19.503973 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:04:19.503925 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-9fbc1b-55c4br4hj9" event={"ID":"fa041808-23c0-4527-aa7f-86e2bfe34e47","Type":"ContainerStarted","Data":"9261f072d7f046f3caa5ace19cb08366f0d18018ac07e90eda32b73daf1cd72e"} Nov 03 03:04:19.504478 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:04:19.503990 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-9fbc1b-55c4br4hj9" event={"ID":"fa041808-23c0-4527-aa7f-86e2bfe34e47","Type":"ContainerStarted","Data":"b41b4f73167d2dfad87a457abc957e707290e34158c96a30ffdb82e6a6888acc"} Nov 03 03:04:19.504478 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:04:19.504018 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-9fbc1b-55c4br4hj9" Nov 03 03:04:19.505923 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:04:19.505899 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-9fbc1b-6df49d566v5qz7" event={"ID":"accac0db-76e1-45ae-b6b3-d637fd501261","Type":"ContainerStarted","Data":"b4a7891f55e4256df09f530ec1aec71797a493a21d534e06cc04de7f32cb8a52"} Nov 03 03:04:19.506044 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:04:19.505940 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-9fbc1b-6df49d566v5qz7" event={"ID":"accac0db-76e1-45ae-b6b3-d637fd501261","Type":"ContainerStarted","Data":"7032106103c4eb0b4badf232f1d55b41abcdc7bf252e86f4d00bf9e09d4913b9"} Nov 03 03:04:19.506044 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:04:19.505991 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-9fbc1b-6df49d566v5qz7" Nov 03 03:04:19.624134 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:04:19.624069 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-9fbc1b-55c4br4hj9" podStartSLOduration=2.624050077 podStartE2EDuration="2.624050077s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:04:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:04:19.622924134 +0000 UTC m=+6341.652406428" watchObservedRunningTime="2026-11-03 03:04:19.624050077 +0000 UTC m=+6341.653532373" Nov 03 03:04:19.743335 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:04:19.743274 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-9fbc1b-6df49d566v5qz7" podStartSLOduration=1.7432584420000001 podStartE2EDuration="1.743258442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:04:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:04:19.741685878 +0000 UTC m=+6341.771168172" watchObservedRunningTime="2026-11-03 03:04:19.743258442 +0000 UTC m=+6341.772740738" Nov 03 03:04:21.346121 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:04:21.346078 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 03:04:21.596772 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:04:21.596677 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 03:04:26.728805 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:04:26.728772 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 03:04:30.512763 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:04:30.512730 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-9fbc1b-55c4br4hj9" Nov 03 03:04:30.513163 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:04:30.512813 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-9fbc1b-6df49d566v5qz7" Nov 03 03:04:32.025336 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:04:32.025301 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 03:04:37.237686 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:04:37.237647 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 03:04:39.143628 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:04:39.143588 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 03:04:39.739001 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:04:39.738966 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 03:04:39.949854 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:04:39.949805 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 03:04:42.335059 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:04:42.335018 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-9fbc1b/simulator-single-main-9fbc1b-kserve-7857f4dc45-5d2p6"] Nov 03 03:04:42.338647 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:04:42.338619 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-9fbc1b/simulator-single-main-9fbc1b-kserve-7857f4dc45-5d2p6" Nov 03 03:04:42.395844 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:04:42.395806 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-9fbc1b\"/\"kube-root-ca.crt\"" Nov 03 03:04:42.417160 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:04:42.417127 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c214c643-5116-49df-ae08-cf076e833127-dshm\") pod \"simulator-single-main-9fbc1b-kserve-7857f4dc45-5d2p6\" (UID: \"c214c643-5116-49df-ae08-cf076e833127\") " pod="e2e-models-e2e-worker-single-main-9fbc1b/simulator-single-main-9fbc1b-kserve-7857f4dc45-5d2p6" Nov 03 03:04:42.417160 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:04:42.417162 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c214c643-5116-49df-ae08-cf076e833127-tmp-dir\") pod \"simulator-single-main-9fbc1b-kserve-7857f4dc45-5d2p6\" (UID: \"c214c643-5116-49df-ae08-cf076e833127\") " pod="e2e-models-e2e-worker-single-main-9fbc1b/simulator-single-main-9fbc1b-kserve-7857f4dc45-5d2p6" Nov 03 03:04:42.417383 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:04:42.417184 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c214c643-5116-49df-ae08-cf076e833127-tls-certs\") pod \"simulator-single-main-9fbc1b-kserve-7857f4dc45-5d2p6\" (UID: \"c214c643-5116-49df-ae08-cf076e833127\") " pod="e2e-models-e2e-worker-single-main-9fbc1b/simulator-single-main-9fbc1b-kserve-7857f4dc45-5d2p6" Nov 03 03:04:42.417383 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:04:42.417307 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c214c643-5116-49df-ae08-cf076e833127-home\") pod \"simulator-single-main-9fbc1b-kserve-7857f4dc45-5d2p6\" (UID: \"c214c643-5116-49df-ae08-cf076e833127\") " pod="e2e-models-e2e-worker-single-main-9fbc1b/simulator-single-main-9fbc1b-kserve-7857f4dc45-5d2p6" Nov 03 03:04:42.417466 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:04:42.417407 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6bmj\" (UniqueName: \"kubernetes.io/projected/c214c643-5116-49df-ae08-cf076e833127-kube-api-access-z6bmj\") pod \"simulator-single-main-9fbc1b-kserve-7857f4dc45-5d2p6\" (UID: \"c214c643-5116-49df-ae08-cf076e833127\") " pod="e2e-models-e2e-worker-single-main-9fbc1b/simulator-single-main-9fbc1b-kserve-7857f4dc45-5d2p6" Nov 03 03:04:42.417466 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:04:42.417453 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c214c643-5116-49df-ae08-cf076e833127-model-cache\") pod \"simulator-single-main-9fbc1b-kserve-7857f4dc45-5d2p6\" (UID: \"c214c643-5116-49df-ae08-cf076e833127\") " pod="e2e-models-e2e-worker-single-main-9fbc1b/simulator-single-main-9fbc1b-kserve-7857f4dc45-5d2p6" Nov 03 03:04:42.446448 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:04:42.446415 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-9fbc1b\"/\"openshift-service-ca.crt\"" Nov 03 03:04:42.446595 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:04:42.446477 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-9fbc1b\"/\"simulator-single-main-9fbc1b-kserve-self-signed-certs\"" Nov 03 03:04:42.446595 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:04:42.446555 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-9fbc1b\"/\"default-dockercfg-mwnrn\"" Nov 03 03:04:42.448503 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:04:42.448484 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-9fbc1b/simulator-single-main-9fbc1b-kserve-7857f4dc45-5d2p6"] Nov 03 03:04:42.500764 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:04:42.500727 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 03:04:42.518245 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:04:42.518214 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6bmj\" (UniqueName: \"kubernetes.io/projected/c214c643-5116-49df-ae08-cf076e833127-kube-api-access-z6bmj\") pod \"simulator-single-main-9fbc1b-kserve-7857f4dc45-5d2p6\" (UID: \"c214c643-5116-49df-ae08-cf076e833127\") " pod="e2e-models-e2e-worker-single-main-9fbc1b/simulator-single-main-9fbc1b-kserve-7857f4dc45-5d2p6" Nov 03 03:04:42.518464 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:04:42.518259 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c214c643-5116-49df-ae08-cf076e833127-model-cache\") pod \"simulator-single-main-9fbc1b-kserve-7857f4dc45-5d2p6\" (UID: \"c214c643-5116-49df-ae08-cf076e833127\") " pod="e2e-models-e2e-worker-single-main-9fbc1b/simulator-single-main-9fbc1b-kserve-7857f4dc45-5d2p6" Nov 03 03:04:42.518464 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:04:42.518308 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c214c643-5116-49df-ae08-cf076e833127-dshm\") pod \"simulator-single-main-9fbc1b-kserve-7857f4dc45-5d2p6\" (UID: \"c214c643-5116-49df-ae08-cf076e833127\") " pod="e2e-models-e2e-worker-single-main-9fbc1b/simulator-single-main-9fbc1b-kserve-7857f4dc45-5d2p6" Nov 03 03:04:42.518464 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:04:42.518325 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c214c643-5116-49df-ae08-cf076e833127-tmp-dir\") pod \"simulator-single-main-9fbc1b-kserve-7857f4dc45-5d2p6\" (UID: \"c214c643-5116-49df-ae08-cf076e833127\") " pod="e2e-models-e2e-worker-single-main-9fbc1b/simulator-single-main-9fbc1b-kserve-7857f4dc45-5d2p6" Nov 03 03:04:42.518464 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:04:42.518346 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c214c643-5116-49df-ae08-cf076e833127-tls-certs\") pod \"simulator-single-main-9fbc1b-kserve-7857f4dc45-5d2p6\" (UID: \"c214c643-5116-49df-ae08-cf076e833127\") " pod="e2e-models-e2e-worker-single-main-9fbc1b/simulator-single-main-9fbc1b-kserve-7857f4dc45-5d2p6" Nov 03 03:04:42.518464 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:04:42.518387 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c214c643-5116-49df-ae08-cf076e833127-home\") pod \"simulator-single-main-9fbc1b-kserve-7857f4dc45-5d2p6\" (UID: \"c214c643-5116-49df-ae08-cf076e833127\") " pod="e2e-models-e2e-worker-single-main-9fbc1b/simulator-single-main-9fbc1b-kserve-7857f4dc45-5d2p6" Nov 03 03:04:42.518860 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:04:42.518765 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c214c643-5116-49df-ae08-cf076e833127-tmp-dir\") pod \"simulator-single-main-9fbc1b-kserve-7857f4dc45-5d2p6\" (UID: \"c214c643-5116-49df-ae08-cf076e833127\") " pod="e2e-models-e2e-worker-single-main-9fbc1b/simulator-single-main-9fbc1b-kserve-7857f4dc45-5d2p6" Nov 03 03:04:42.518860 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:04:42.518791 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c214c643-5116-49df-ae08-cf076e833127-model-cache\") pod \"simulator-single-main-9fbc1b-kserve-7857f4dc45-5d2p6\" (UID: \"c214c643-5116-49df-ae08-cf076e833127\") " pod="e2e-models-e2e-worker-single-main-9fbc1b/simulator-single-main-9fbc1b-kserve-7857f4dc45-5d2p6" Nov 03 03:04:42.518860 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:04:42.518842 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c214c643-5116-49df-ae08-cf076e833127-home\") pod \"simulator-single-main-9fbc1b-kserve-7857f4dc45-5d2p6\" (UID: \"c214c643-5116-49df-ae08-cf076e833127\") " pod="e2e-models-e2e-worker-single-main-9fbc1b/simulator-single-main-9fbc1b-kserve-7857f4dc45-5d2p6" Nov 03 03:04:42.520788 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:04:42.520764 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c214c643-5116-49df-ae08-cf076e833127-dshm\") pod \"simulator-single-main-9fbc1b-kserve-7857f4dc45-5d2p6\" (UID: \"c214c643-5116-49df-ae08-cf076e833127\") " pod="e2e-models-e2e-worker-single-main-9fbc1b/simulator-single-main-9fbc1b-kserve-7857f4dc45-5d2p6" Nov 03 03:04:42.521198 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:04:42.521177 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c214c643-5116-49df-ae08-cf076e833127-tls-certs\") pod \"simulator-single-main-9fbc1b-kserve-7857f4dc45-5d2p6\" (UID: \"c214c643-5116-49df-ae08-cf076e833127\") " pod="e2e-models-e2e-worker-single-main-9fbc1b/simulator-single-main-9fbc1b-kserve-7857f4dc45-5d2p6" Nov 03 03:04:42.590628 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:04:42.590555 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6bmj\" (UniqueName: \"kubernetes.io/projected/c214c643-5116-49df-ae08-cf076e833127-kube-api-access-z6bmj\") pod \"simulator-single-main-9fbc1b-kserve-7857f4dc45-5d2p6\" (UID: \"c214c643-5116-49df-ae08-cf076e833127\") " pod="e2e-models-e2e-worker-single-main-9fbc1b/simulator-single-main-9fbc1b-kserve-7857f4dc45-5d2p6" Nov 03 03:04:42.654312 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:04:42.654277 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-9fbc1b/simulator-single-main-9fbc1b-kserve-7857f4dc45-5d2p6" Nov 03 03:04:43.042267 ip-10-0-136-16 kubenswrapper[5437]: W1103 03:04:43.042228 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc214c643_5116_49df_ae08_cf076e833127.slice/crio-5488b74585c8b043991d034edaf73c3fc16abbfb994ea5b484048b50198444cc WatchSource:0}: Error finding container 5488b74585c8b043991d034edaf73c3fc16abbfb994ea5b484048b50198444cc: Status 404 returned error can't find the container with id 5488b74585c8b043991d034edaf73c3fc16abbfb994ea5b484048b50198444cc Nov 03 03:04:43.043970 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:04:43.043944 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-9fbc1b/simulator-single-main-9fbc1b-kserve-7857f4dc45-5d2p6"] Nov 03 03:04:43.057744 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:04:43.057714 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 03:04:43.633200 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:04:43.633157 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-9fbc1b/simulator-single-main-9fbc1b-kserve-7857f4dc45-5d2p6" event={"ID":"c214c643-5116-49df-ae08-cf076e833127","Type":"ContainerStarted","Data":"7a976bf386d7649910728c34e1a4c12dae18d79afc638b5b4793a6adda8f1a90"} Nov 03 03:04:43.633640 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:04:43.633211 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-9fbc1b/simulator-single-main-9fbc1b-kserve-7857f4dc45-5d2p6" event={"ID":"c214c643-5116-49df-ae08-cf076e833127","Type":"ContainerStarted","Data":"5488b74585c8b043991d034edaf73c3fc16abbfb994ea5b484048b50198444cc"} Nov 03 03:04:43.654423 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:04:43.654390 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-9fbc1b/simulator-single-main-9fbc1b-kserve-7857f4dc45-5d2p6" Nov 03 03:04:44.034968 ip-10-0-136-16 kubenswrapper[5437]: E1103 03:04:44.034888 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 03:04:48.309984 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:04:48.309945 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 03:04:52.655335 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:04:52.655300 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-9fbc1b/simulator-single-main-9fbc1b-kserve-7857f4dc45-5d2p6" Nov 03 03:04:52.667953 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:04:52.667926 5437 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-9fbc1b/simulator-single-main-9fbc1b-kserve-7857f4dc45-5d2p6" Nov 03 03:04:52.689047 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:04:52.689019 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-9fbc1b/simulator-single-main-9fbc1b-kserve-7857f4dc45-5d2p6" Nov 03 03:04:52.792388 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:04:52.792308 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-9fbc1b/simulator-single-main-9fbc1b-kserve-7857f4dc45-5d2p6" podStartSLOduration=10.792295299 podStartE2EDuration="10.792295299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:04:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:04:43.760125014 +0000 UTC m=+6365.789607310" watchObservedRunningTime="2026-11-03 03:04:52.792295299 +0000 UTC m=+6374.821777594" Nov 03 03:04:53.749871 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:04:53.749822 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 03:04:59.070563 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:04:59.070516 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 03:05:04.414600 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:05:04.414565 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 03:05:09.725502 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:05:09.725450 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 03:05:12.391702 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:05:12.391655 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 03:05:12.946402 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:05:12.946349 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 03:05:14.034879 ip-10-0-136-16 kubenswrapper[5437]: E1103 03:05:14.034844 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 03:05:15.029429 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:05:15.029387 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 03:05:17.208006 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:05:17.207912 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 03:05:20.334030 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:05:20.333991 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 03:05:24.527285 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:05:24.527238 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 03:05:25.628884 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:05:25.628836 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 03:05:30.858695 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:05:30.858648 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 03:05:36.129574 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:05:36.129532 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 03:05:41.320448 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:05:41.320407 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 03:05:44.034916 ip-10-0-136-16 kubenswrapper[5437]: E1103 03:05:44.034871 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 03:05:44.410820 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:05:44.410778 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 03:05:44.895792 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:05:44.895756 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 03:05:46.622710 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:05:46.622615 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 03:05:49.324052 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:05:49.324017 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-9fbc1b/simulator-single-main-9fbc1b-kserve-7857f4dc45-5d2p6"] Nov 03 03:05:49.461961 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:05:49.461919 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 03:05:49.964469 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:05:49.964414 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-9fbc1b-6df49d566v5qz7"] Nov 03 03:05:50.071706 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:05:50.071669 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-9fbc1b-55c4br4hj9"] Nov 03 03:05:50.374710 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:05:50.374673 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 03:05:50.632900 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:05:50.632805 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 03:05:51.221242 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:05:51.221206 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 03:05:52.147314 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:05:52.147281 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 03:05:54.965152 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:05:54.965082 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-9fbc1b-6df49d566v5qz7" podUID="accac0db-76e1-45ae-b6b3-d637fd501261" containerName="payload-processing" containerID="cri-o://b4a7891f55e4256df09f530ec1aec71797a493a21d534e06cc04de7f32cb8a52" gracePeriod=25 Nov 03 03:05:55.072418 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:05:55.072350 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-9fbc1b-55c4br4hj9" podUID="fa041808-23c0-4527-aa7f-86e2bfe34e47" containerName="payload-pre-processing" containerID="cri-o://9261f072d7f046f3caa5ace19cb08366f0d18018ac07e90eda32b73daf1cd72e" gracePeriod=25 Nov 03 03:05:56.903765 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:05:56.903731 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 03:05:57.151449 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:05:57.151415 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 03:06:00.515386 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:06:00.512080 5437 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-9fbc1b-6df49d566v5qz7" podUID="accac0db-76e1-45ae-b6b3-d637fd501261" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 03:06:00.519132 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:06:00.519099 5437 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-9fbc1b-55c4br4hj9" podUID="fa041808-23c0-4527-aa7f-86e2bfe34e47" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 03:06:02.523523 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:06:02.523448 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 03:06:02.987593 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:06:02.987553 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 03:06:04.345060 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:06:04.345001 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-9fbc1b/simulator-single-main-9fbc1b-kserve-7857f4dc45-5d2p6" podUID="c214c643-5116-49df-ae08-cf076e833127" containerName="main" containerID="cri-o://7a976bf386d7649910728c34e1a4c12dae18d79afc638b5b4793a6adda8f1a90" gracePeriod=45 Nov 03 03:06:04.594581 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:06:04.594554 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-9fbc1b/simulator-single-main-9fbc1b-kserve-7857f4dc45-5d2p6" Nov 03 03:06:04.737929 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:06:04.737843 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c214c643-5116-49df-ae08-cf076e833127-tls-certs\" (UniqueName: \"kubernetes.io/secret/c214c643-5116-49df-ae08-cf076e833127-tls-certs\") pod \"c214c643-5116-49df-ae08-cf076e833127\" (UID: \"c214c643-5116-49df-ae08-cf076e833127\") " Nov 03 03:06:04.738113 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:06:04.737952 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c214c643-5116-49df-ae08-cf076e833127-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c214c643-5116-49df-ae08-cf076e833127-model-cache\") pod \"c214c643-5116-49df-ae08-cf076e833127\" (UID: \"c214c643-5116-49df-ae08-cf076e833127\") " Nov 03 03:06:04.738113 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:06:04.737984 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c214c643-5116-49df-ae08-cf076e833127-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c214c643-5116-49df-ae08-cf076e833127-dshm\") pod \"c214c643-5116-49df-ae08-cf076e833127\" (UID: \"c214c643-5116-49df-ae08-cf076e833127\") " Nov 03 03:06:04.738113 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:06:04.738070 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c214c643-5116-49df-ae08-cf076e833127-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c214c643-5116-49df-ae08-cf076e833127-tmp-dir\") pod \"c214c643-5116-49df-ae08-cf076e833127\" (UID: \"c214c643-5116-49df-ae08-cf076e833127\") " Nov 03 03:06:04.738113 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:06:04.738104 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c214c643-5116-49df-ae08-cf076e833127-home\" (UniqueName: \"kubernetes.io/empty-dir/c214c643-5116-49df-ae08-cf076e833127-home\") pod \"c214c643-5116-49df-ae08-cf076e833127\" (UID: \"c214c643-5116-49df-ae08-cf076e833127\") " Nov 03 03:06:04.738351 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:06:04.738144 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c214c643-5116-49df-ae08-cf076e833127-kube-api-access-z6bmj\" (UniqueName: \"kubernetes.io/projected/c214c643-5116-49df-ae08-cf076e833127-kube-api-access-z6bmj\") pod \"c214c643-5116-49df-ae08-cf076e833127\" (UID: \"c214c643-5116-49df-ae08-cf076e833127\") " Nov 03 03:06:04.738454 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:06:04.738392 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c214c643-5116-49df-ae08-cf076e833127-tmp-dir" pod "c214c643-5116-49df-ae08-cf076e833127" (UID: "c214c643-5116-49df-ae08-cf076e833127"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:06:04.738454 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:06:04.738424 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c214c643-5116-49df-ae08-cf076e833127-home" pod "c214c643-5116-49df-ae08-cf076e833127" (UID: "c214c643-5116-49df-ae08-cf076e833127"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:06:04.738567 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:06:04.738489 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c214c643-5116-49df-ae08-cf076e833127-model-cache" pod "c214c643-5116-49df-ae08-cf076e833127" (UID: "c214c643-5116-49df-ae08-cf076e833127"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:06:04.738567 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:06:04.738543 5437 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c214c643-5116-49df-ae08-cf076e833127-tmp-dir\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:06:04.738567 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:06:04.738555 5437 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c214c643-5116-49df-ae08-cf076e833127-home\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:06:04.745399 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:06:04.740593 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c214c643-5116-49df-ae08-cf076e833127-tls-certs" pod "c214c643-5116-49df-ae08-cf076e833127" (UID: "c214c643-5116-49df-ae08-cf076e833127"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:06:04.745399 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:06:04.740751 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c214c643-5116-49df-ae08-cf076e833127-kube-api-access-z6bmj" pod "c214c643-5116-49df-ae08-cf076e833127" (UID: "c214c643-5116-49df-ae08-cf076e833127"). InnerVolumeSpecName "kube-api-access-z6bmj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:06:04.745399 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:06:04.741097 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c214c643-5116-49df-ae08-cf076e833127-dshm" pod "c214c643-5116-49df-ae08-cf076e833127" (UID: "c214c643-5116-49df-ae08-cf076e833127"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:06:04.839251 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:06:04.839206 5437 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c214c643-5116-49df-ae08-cf076e833127-model-cache\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:06:04.839251 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:06:04.839243 5437 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c214c643-5116-49df-ae08-cf076e833127-dshm\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:06:04.839537 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:06:04.839258 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6bmj\" (UniqueName: \"kubernetes.io/projected/c214c643-5116-49df-ae08-cf076e833127-kube-api-access-z6bmj\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:06:04.839537 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:06:04.839274 5437 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c214c643-5116-49df-ae08-cf076e833127-tls-certs\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:06:05.038272 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:06:05.038182 5437 generic.go:356] "Generic (PLEG): container finished" podID="c214c643-5116-49df-ae08-cf076e833127" containerID="7a976bf386d7649910728c34e1a4c12dae18d79afc638b5b4793a6adda8f1a90" exitCode=0 Nov 03 03:06:05.038272 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:06:05.038233 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-9fbc1b/simulator-single-main-9fbc1b-kserve-7857f4dc45-5d2p6" Nov 03 03:06:05.038272 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:06:05.038255 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-9fbc1b/simulator-single-main-9fbc1b-kserve-7857f4dc45-5d2p6" event={"ID":"c214c643-5116-49df-ae08-cf076e833127","Type":"ContainerDied","Data":"7a976bf386d7649910728c34e1a4c12dae18d79afc638b5b4793a6adda8f1a90"} Nov 03 03:06:05.038610 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:06:05.038296 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-9fbc1b/simulator-single-main-9fbc1b-kserve-7857f4dc45-5d2p6" event={"ID":"c214c643-5116-49df-ae08-cf076e833127","Type":"ContainerDied","Data":"5488b74585c8b043991d034edaf73c3fc16abbfb994ea5b484048b50198444cc"} Nov 03 03:06:05.038610 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:06:05.038312 5437 scope.go:122] "RemoveContainer" containerID="7a976bf386d7649910728c34e1a4c12dae18d79afc638b5b4793a6adda8f1a90" Nov 03 03:06:05.048012 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:06:05.047991 5437 scope.go:122] "RemoveContainer" containerID="7a976bf386d7649910728c34e1a4c12dae18d79afc638b5b4793a6adda8f1a90" Nov 03 03:06:05.048324 ip-10-0-136-16 kubenswrapper[5437]: E1103 03:06:05.048304 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a976bf386d7649910728c34e1a4c12dae18d79afc638b5b4793a6adda8f1a90\": container with ID starting with 7a976bf386d7649910728c34e1a4c12dae18d79afc638b5b4793a6adda8f1a90 not found: ID does not exist" containerID="7a976bf386d7649910728c34e1a4c12dae18d79afc638b5b4793a6adda8f1a90" Nov 03 03:06:05.048410 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:06:05.048336 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a976bf386d7649910728c34e1a4c12dae18d79afc638b5b4793a6adda8f1a90"} err="failed to get container status \"7a976bf386d7649910728c34e1a4c12dae18d79afc638b5b4793a6adda8f1a90\": rpc error: code = NotFound desc = could not find container \"7a976bf386d7649910728c34e1a4c12dae18d79afc638b5b4793a6adda8f1a90\": container with ID starting with 7a976bf386d7649910728c34e1a4c12dae18d79afc638b5b4793a6adda8f1a90 not found: ID does not exist" Nov 03 03:06:05.207830 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:06:05.207794 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-9fbc1b/simulator-single-main-9fbc1b-kserve-7857f4dc45-5d2p6"] Nov 03 03:06:05.259163 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:06:05.259125 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-9fbc1b/simulator-single-main-9fbc1b-kserve-7857f4dc45-5d2p6"] Nov 03 03:06:06.087619 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:06:06.087582 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c214c643-5116-49df-ae08-cf076e833127" path="/var/lib/kubelet/pods/c214c643-5116-49df-ae08-cf076e833127/volumes" Nov 03 03:06:10.512150 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:06:10.512107 5437 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-9fbc1b-55c4br4hj9" podUID="fa041808-23c0-4527-aa7f-86e2bfe34e47" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 03:06:10.512598 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:06:10.512107 5437 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-9fbc1b-6df49d566v5qz7" podUID="accac0db-76e1-45ae-b6b3-d637fd501261" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 03:06:12.680175 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:06:12.680139 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 03:06:14.033979 ip-10-0-136-16 kubenswrapper[5437]: E1103 03:06:14.033941 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 03:06:15.090065 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:06:15.090038 5437 generic.go:356] "Generic (PLEG): container finished" podID="accac0db-76e1-45ae-b6b3-d637fd501261" containerID="b4a7891f55e4256df09f530ec1aec71797a493a21d534e06cc04de7f32cb8a52" exitCode=0 Nov 03 03:06:15.090422 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:06:15.090112 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-9fbc1b-6df49d566v5qz7" event={"ID":"accac0db-76e1-45ae-b6b3-d637fd501261","Type":"ContainerDied","Data":"b4a7891f55e4256df09f530ec1aec71797a493a21d534e06cc04de7f32cb8a52"} Nov 03 03:06:15.217831 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:06:15.217799 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-9fbc1b-6df49d566v5qz7" Nov 03 03:06:15.317047 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:06:15.317020 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-9fbc1b-55c4br4hj9" Nov 03 03:06:15.345247 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:06:15.345219 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/accac0db-76e1-45ae-b6b3-d637fd501261-kube-api-access-2ngtt\" (UniqueName: \"kubernetes.io/projected/accac0db-76e1-45ae-b6b3-d637fd501261-kube-api-access-2ngtt\") pod \"accac0db-76e1-45ae-b6b3-d637fd501261\" (UID: \"accac0db-76e1-45ae-b6b3-d637fd501261\") " Nov 03 03:06:15.345427 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:06:15.345412 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/accac0db-76e1-45ae-b6b3-d637fd501261-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/accac0db-76e1-45ae-b6b3-d637fd501261-plugins-config-volume\") pod \"accac0db-76e1-45ae-b6b3-d637fd501261\" (UID: \"accac0db-76e1-45ae-b6b3-d637fd501261\") " Nov 03 03:06:15.345782 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:06:15.345752 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/accac0db-76e1-45ae-b6b3-d637fd501261-plugins-config-volume" pod "accac0db-76e1-45ae-b6b3-d637fd501261" (UID: "accac0db-76e1-45ae-b6b3-d637fd501261"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:06:15.347269 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:06:15.347247 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/accac0db-76e1-45ae-b6b3-d637fd501261-kube-api-access-2ngtt" pod "accac0db-76e1-45ae-b6b3-d637fd501261" (UID: "accac0db-76e1-45ae-b6b3-d637fd501261"). InnerVolumeSpecName "kube-api-access-2ngtt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:06:15.446734 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:06:15.446700 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa041808-23c0-4527-aa7f-86e2bfe34e47-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa041808-23c0-4527-aa7f-86e2bfe34e47-plugins-config-volume\") pod \"fa041808-23c0-4527-aa7f-86e2bfe34e47\" (UID: \"fa041808-23c0-4527-aa7f-86e2bfe34e47\") " Nov 03 03:06:15.446933 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:06:15.446752 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa041808-23c0-4527-aa7f-86e2bfe34e47-kube-api-access-689bt\" (UniqueName: \"kubernetes.io/projected/fa041808-23c0-4527-aa7f-86e2bfe34e47-kube-api-access-689bt\") pod \"fa041808-23c0-4527-aa7f-86e2bfe34e47\" (UID: \"fa041808-23c0-4527-aa7f-86e2bfe34e47\") " Nov 03 03:06:15.447045 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:06:15.447027 5437 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/accac0db-76e1-45ae-b6b3-d637fd501261-plugins-config-volume\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:06:15.447101 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:06:15.447039 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa041808-23c0-4527-aa7f-86e2bfe34e47-plugins-config-volume" pod "fa041808-23c0-4527-aa7f-86e2bfe34e47" (UID: "fa041808-23c0-4527-aa7f-86e2bfe34e47"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:06:15.447101 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:06:15.447052 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ngtt\" (UniqueName: \"kubernetes.io/projected/accac0db-76e1-45ae-b6b3-d637fd501261-kube-api-access-2ngtt\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:06:15.448827 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:06:15.448804 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa041808-23c0-4527-aa7f-86e2bfe34e47-kube-api-access-689bt" pod "fa041808-23c0-4527-aa7f-86e2bfe34e47" (UID: "fa041808-23c0-4527-aa7f-86e2bfe34e47"). InnerVolumeSpecName "kube-api-access-689bt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:06:15.548470 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:06:15.548427 5437 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa041808-23c0-4527-aa7f-86e2bfe34e47-plugins-config-volume\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:06:15.548470 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:06:15.548469 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-689bt\" (UniqueName: \"kubernetes.io/projected/fa041808-23c0-4527-aa7f-86e2bfe34e47-kube-api-access-689bt\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:06:16.103582 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:06:16.103528 5437 generic.go:356] "Generic (PLEG): container finished" podID="fa041808-23c0-4527-aa7f-86e2bfe34e47" containerID="9261f072d7f046f3caa5ace19cb08366f0d18018ac07e90eda32b73daf1cd72e" exitCode=0 Nov 03 03:06:16.103964 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:06:16.103605 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-9fbc1b-55c4br4hj9" Nov 03 03:06:16.103964 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:06:16.103605 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-9fbc1b-55c4br4hj9" event={"ID":"fa041808-23c0-4527-aa7f-86e2bfe34e47","Type":"ContainerDied","Data":"9261f072d7f046f3caa5ace19cb08366f0d18018ac07e90eda32b73daf1cd72e"} Nov 03 03:06:16.103964 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:06:16.103677 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-9fbc1b-55c4br4hj9" event={"ID":"fa041808-23c0-4527-aa7f-86e2bfe34e47","Type":"ContainerDied","Data":"b41b4f73167d2dfad87a457abc957e707290e34158c96a30ffdb82e6a6888acc"} Nov 03 03:06:16.103964 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:06:16.103702 5437 scope.go:122] "RemoveContainer" containerID="9261f072d7f046f3caa5ace19cb08366f0d18018ac07e90eda32b73daf1cd72e" Nov 03 03:06:16.105068 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:06:16.105046 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-9fbc1b-6df49d566v5qz7" event={"ID":"accac0db-76e1-45ae-b6b3-d637fd501261","Type":"ContainerDied","Data":"7032106103c4eb0b4badf232f1d55b41abcdc7bf252e86f4d00bf9e09d4913b9"} Nov 03 03:06:16.105130 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:06:16.105075 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-9fbc1b-6df49d566v5qz7" Nov 03 03:06:16.113035 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:06:16.113018 5437 scope.go:122] "RemoveContainer" containerID="9261f072d7f046f3caa5ace19cb08366f0d18018ac07e90eda32b73daf1cd72e" Nov 03 03:06:16.113322 ip-10-0-136-16 kubenswrapper[5437]: E1103 03:06:16.113305 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9261f072d7f046f3caa5ace19cb08366f0d18018ac07e90eda32b73daf1cd72e\": container with ID starting with 9261f072d7f046f3caa5ace19cb08366f0d18018ac07e90eda32b73daf1cd72e not found: ID does not exist" containerID="9261f072d7f046f3caa5ace19cb08366f0d18018ac07e90eda32b73daf1cd72e" Nov 03 03:06:16.113380 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:06:16.113329 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9261f072d7f046f3caa5ace19cb08366f0d18018ac07e90eda32b73daf1cd72e"} err="failed to get container status \"9261f072d7f046f3caa5ace19cb08366f0d18018ac07e90eda32b73daf1cd72e\": rpc error: code = NotFound desc = could not find container \"9261f072d7f046f3caa5ace19cb08366f0d18018ac07e90eda32b73daf1cd72e\": container with ID starting with 9261f072d7f046f3caa5ace19cb08366f0d18018ac07e90eda32b73daf1cd72e not found: ID does not exist" Nov 03 03:06:16.113380 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:06:16.113346 5437 scope.go:122] "RemoveContainer" containerID="b4a7891f55e4256df09f530ec1aec71797a493a21d534e06cc04de7f32cb8a52" Nov 03 03:06:16.272680 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:06:16.272603 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-9fbc1b-6df49d566v5qz7"] Nov 03 03:06:16.324543 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:06:16.324508 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-9fbc1b-6df49d566v5qz7"] Nov 03 03:06:16.441181 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:06:16.441146 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-9fbc1b-55c4br4hj9"] Nov 03 03:06:16.444832 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:06:16.444804 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-9fbc1b-55c4br4hj9"] Nov 03 03:06:17.561514 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:06:17.561397 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 03:06:18.087212 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:06:18.087180 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="accac0db-76e1-45ae-b6b3-d637fd501261" path="/var/lib/kubelet/pods/accac0db-76e1-45ae-b6b3-d637fd501261/volumes" Nov 03 03:06:18.087612 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:06:18.087596 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa041808-23c0-4527-aa7f-86e2bfe34e47" path="/var/lib/kubelet/pods/fa041808-23c0-4527-aa7f-86e2bfe34e47/volumes" Nov 03 03:06:26.008838 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:06:26.008797 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 03:06:26.855599 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:06:26.855560 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 03:06:28.819562 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:06:28.819521 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 03:06:29.029419 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:06:29.029360 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 03:06:30.273773 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:06:30.273735 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 03:06:30.611447 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:06:30.611397 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 03:06:31.246919 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:06:31.246882 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 03:06:32.917171 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:06:32.917122 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 03:06:33.125893 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:06:33.125856 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 03:06:44.033644 ip-10-0-136-16 kubenswrapper[5437]: E1103 03:06:44.033596 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 03:07:13.038781 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:07:13.038696 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 03:07:14.034429 ip-10-0-136-16 kubenswrapper[5437]: E1103 03:07:14.034394 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 03:07:17.905634 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:07:17.905600 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 03:07:44.034093 ip-10-0-136-16 kubenswrapper[5437]: E1103 03:07:44.034045 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 03:08:13.384402 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:08:13.384334 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 03:08:14.034052 ip-10-0-136-16 kubenswrapper[5437]: E1103 03:08:14.034009 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 03:08:18.329446 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:08:18.326946 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 03:08:39.525390 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:08:39.522789 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 03:08:44.036503 ip-10-0-136-16 kubenswrapper[5437]: E1103 03:08:44.036457 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 03:08:55.394716 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:08:55.394682 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-67759c8968-ht5tg"] Nov 03 03:08:55.395224 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:08:55.395178 5437 cpu_manager.go:414] "containerMap: removing container" podUID="fa041808-23c0-4527-aa7f-86e2bfe34e47" containerName="payload-pre-processing" Nov 03 03:08:55.395224 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:08:55.395197 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa041808-23c0-4527-aa7f-86e2bfe34e47" containerName="payload-pre-processing" Nov 03 03:08:55.395344 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:08:55.395234 5437 cpu_manager.go:414] "containerMap: removing container" podUID="c214c643-5116-49df-ae08-cf076e833127" containerName="main" Nov 03 03:08:55.395344 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:08:55.395242 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c214c643-5116-49df-ae08-cf076e833127" containerName="main" Nov 03 03:08:55.395344 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:08:55.395261 5437 cpu_manager.go:414] "containerMap: removing container" podUID="accac0db-76e1-45ae-b6b3-d637fd501261" containerName="payload-processing" Nov 03 03:08:55.395344 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:08:55.395267 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="accac0db-76e1-45ae-b6b3-d637fd501261" containerName="payload-processing" Nov 03 03:08:55.395535 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:08:55.395408 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa041808-23c0-4527-aa7f-86e2bfe34e47" containerName="payload-pre-processing" Nov 03 03:08:55.395535 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:08:55.395427 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="c214c643-5116-49df-ae08-cf076e833127" containerName="main" Nov 03 03:08:55.395535 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:08:55.395447 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="accac0db-76e1-45ae-b6b3-d637fd501261" containerName="payload-processing" Nov 03 03:08:55.398762 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:08:55.398744 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-67759c8968-ht5tg" Nov 03 03:08:55.470692 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:08:55.470657 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1edae360-1d25-4785-b969-f27cf0fe5c5e-maas-api-tls\") pod \"maas-api-67759c8968-ht5tg\" (UID: \"1edae360-1d25-4785-b969-f27cf0fe5c5e\") " pod="odh-ai-gateway-infra/maas-api-67759c8968-ht5tg" Nov 03 03:08:55.470993 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:08:55.470961 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1edae360-1d25-4785-b969-f27cf0fe5c5e-openshift-service-ca\") pod \"maas-api-67759c8968-ht5tg\" (UID: \"1edae360-1d25-4785-b969-f27cf0fe5c5e\") " pod="odh-ai-gateway-infra/maas-api-67759c8968-ht5tg" Nov 03 03:08:55.471118 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:08:55.471021 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1edae360-1d25-4785-b969-f27cf0fe5c5e-metrics-certs\") pod \"maas-api-67759c8968-ht5tg\" (UID: \"1edae360-1d25-4785-b969-f27cf0fe5c5e\") " pod="odh-ai-gateway-infra/maas-api-67759c8968-ht5tg" Nov 03 03:08:55.471229 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:08:55.471170 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qq6pb\" (UniqueName: \"kubernetes.io/projected/1edae360-1d25-4785-b969-f27cf0fe5c5e-kube-api-access-qq6pb\") pod \"maas-api-67759c8968-ht5tg\" (UID: \"1edae360-1d25-4785-b969-f27cf0fe5c5e\") " pod="odh-ai-gateway-infra/maas-api-67759c8968-ht5tg" Nov 03 03:08:55.505965 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:08:55.505928 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-67759c8968-ht5tg"] Nov 03 03:08:55.572689 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:08:55.572647 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1edae360-1d25-4785-b969-f27cf0fe5c5e-maas-api-tls\") pod \"maas-api-67759c8968-ht5tg\" (UID: \"1edae360-1d25-4785-b969-f27cf0fe5c5e\") " pod="odh-ai-gateway-infra/maas-api-67759c8968-ht5tg" Nov 03 03:08:55.572902 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:08:55.572722 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1edae360-1d25-4785-b969-f27cf0fe5c5e-openshift-service-ca\") pod \"maas-api-67759c8968-ht5tg\" (UID: \"1edae360-1d25-4785-b969-f27cf0fe5c5e\") " pod="odh-ai-gateway-infra/maas-api-67759c8968-ht5tg" Nov 03 03:08:55.572902 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:08:55.572777 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1edae360-1d25-4785-b969-f27cf0fe5c5e-metrics-certs\") pod \"maas-api-67759c8968-ht5tg\" (UID: \"1edae360-1d25-4785-b969-f27cf0fe5c5e\") " pod="odh-ai-gateway-infra/maas-api-67759c8968-ht5tg" Nov 03 03:08:55.572902 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:08:55.572871 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qq6pb\" (UniqueName: \"kubernetes.io/projected/1edae360-1d25-4785-b969-f27cf0fe5c5e-kube-api-access-qq6pb\") pod \"maas-api-67759c8968-ht5tg\" (UID: \"1edae360-1d25-4785-b969-f27cf0fe5c5e\") " pod="odh-ai-gateway-infra/maas-api-67759c8968-ht5tg" Nov 03 03:08:55.573456 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:08:55.573431 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1edae360-1d25-4785-b969-f27cf0fe5c5e-openshift-service-ca\") pod \"maas-api-67759c8968-ht5tg\" (UID: \"1edae360-1d25-4785-b969-f27cf0fe5c5e\") " pod="odh-ai-gateway-infra/maas-api-67759c8968-ht5tg" Nov 03 03:08:55.575247 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:08:55.575221 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1edae360-1d25-4785-b969-f27cf0fe5c5e-metrics-certs\") pod \"maas-api-67759c8968-ht5tg\" (UID: \"1edae360-1d25-4785-b969-f27cf0fe5c5e\") " pod="odh-ai-gateway-infra/maas-api-67759c8968-ht5tg" Nov 03 03:08:55.575340 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:08:55.575242 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1edae360-1d25-4785-b969-f27cf0fe5c5e-maas-api-tls\") pod \"maas-api-67759c8968-ht5tg\" (UID: \"1edae360-1d25-4785-b969-f27cf0fe5c5e\") " pod="odh-ai-gateway-infra/maas-api-67759c8968-ht5tg" Nov 03 03:08:55.635981 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:08:55.635946 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qq6pb\" (UniqueName: \"kubernetes.io/projected/1edae360-1d25-4785-b969-f27cf0fe5c5e-kube-api-access-qq6pb\") pod \"maas-api-67759c8968-ht5tg\" (UID: \"1edae360-1d25-4785-b969-f27cf0fe5c5e\") " pod="odh-ai-gateway-infra/maas-api-67759c8968-ht5tg" Nov 03 03:08:55.712914 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:08:55.712822 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-67759c8968-ht5tg" Nov 03 03:08:56.082725 ip-10-0-136-16 kubenswrapper[5437]: W1103 03:08:56.082692 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1edae360_1d25_4785_b969_f27cf0fe5c5e.slice/crio-027305a0fc2332435d30a891e22a4aee89f50a7e035c7302d663138008a2cdd4 WatchSource:0}: Error finding container 027305a0fc2332435d30a891e22a4aee89f50a7e035c7302d663138008a2cdd4: Status 404 returned error can't find the container with id 027305a0fc2332435d30a891e22a4aee89f50a7e035c7302d663138008a2cdd4 Nov 03 03:08:56.084186 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:08:56.084160 5437 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 03:08:56.088913 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:08:56.088891 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-67759c8968-ht5tg"] Nov 03 03:08:56.885610 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:08:56.885563 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-67759c8968-ht5tg" event={"ID":"1edae360-1d25-4785-b969-f27cf0fe5c5e","Type":"ContainerStarted","Data":"027305a0fc2332435d30a891e22a4aee89f50a7e035c7302d663138008a2cdd4"} Nov 03 03:08:57.893246 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:08:57.893204 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-67759c8968-ht5tg" event={"ID":"1edae360-1d25-4785-b969-f27cf0fe5c5e","Type":"ContainerStarted","Data":"82059b03ed89089fa7822b0d82671357aada672d8562a7b5307b8b1e2b9fd504"} Nov 03 03:08:57.893699 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:08:57.893353 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-67759c8968-ht5tg" Nov 03 03:08:58.016899 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:08:58.016849 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-67759c8968-ht5tg" podStartSLOduration=2.042183018 podStartE2EDuration="3.016832143s" totalImagesPullingTime="974.649125ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:08:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:08:58.015059264 +0000 UTC m=+6620.044541559" watchObservedRunningTime="2026-11-03 03:08:58.016832143 +0000 UTC m=+6620.046314434" Nov 03 03:09:03.903345 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:09:03.903313 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-67759c8968-ht5tg" Nov 03 03:09:04.078742 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:09:04.078705 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6b6d569d44-rm7ch"] Nov 03 03:09:04.078958 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:09:04.078932 5437 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6b6d569d44-rm7ch" podUID="e301e8c2-2042-4641-a416-e26c636c2dfd" containerName="maas-api" containerID="cri-o://9e0f91fdc0e28715bc88d22d937818a08b3a2a74ef18c7f12fe9f0fe1961a71e" gracePeriod=30 Nov 03 03:09:04.708877 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:09:04.708831 5437 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-6b6d569d44-rm7ch" podUID="e301e8c2-2042-4641-a416-e26c636c2dfd" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.28:8443/health\": dial tcp 10.134.0.28:8443: connect: connection refused" Nov 03 03:09:05.339208 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:09:05.339177 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6b6d569d44-rm7ch" Nov 03 03:09:05.472922 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:09:05.472844 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e301e8c2-2042-4641-a416-e26c636c2dfd-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e301e8c2-2042-4641-a416-e26c636c2dfd-maas-api-tls\") pod \"e301e8c2-2042-4641-a416-e26c636c2dfd\" (UID: \"e301e8c2-2042-4641-a416-e26c636c2dfd\") " Nov 03 03:09:05.472922 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:09:05.472910 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e301e8c2-2042-4641-a416-e26c636c2dfd-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e301e8c2-2042-4641-a416-e26c636c2dfd-openshift-service-ca\") pod \"e301e8c2-2042-4641-a416-e26c636c2dfd\" (UID: \"e301e8c2-2042-4641-a416-e26c636c2dfd\") " Nov 03 03:09:05.473162 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:09:05.472965 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e301e8c2-2042-4641-a416-e26c636c2dfd-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e301e8c2-2042-4641-a416-e26c636c2dfd-metrics-certs\") pod \"e301e8c2-2042-4641-a416-e26c636c2dfd\" (UID: \"e301e8c2-2042-4641-a416-e26c636c2dfd\") " Nov 03 03:09:05.473162 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:09:05.473057 5437 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e301e8c2-2042-4641-a416-e26c636c2dfd-kube-api-access-z9c8q\" (UniqueName: \"kubernetes.io/projected/e301e8c2-2042-4641-a416-e26c636c2dfd-kube-api-access-z9c8q\") pod \"e301e8c2-2042-4641-a416-e26c636c2dfd\" (UID: \"e301e8c2-2042-4641-a416-e26c636c2dfd\") " Nov 03 03:09:05.473312 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:09:05.473286 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e301e8c2-2042-4641-a416-e26c636c2dfd-openshift-service-ca" pod "e301e8c2-2042-4641-a416-e26c636c2dfd" (UID: "e301e8c2-2042-4641-a416-e26c636c2dfd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:09:05.473486 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:09:05.473466 5437 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e301e8c2-2042-4641-a416-e26c636c2dfd-openshift-service-ca\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:09:05.475214 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:09:05.475186 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e301e8c2-2042-4641-a416-e26c636c2dfd-maas-api-tls" pod "e301e8c2-2042-4641-a416-e26c636c2dfd" (UID: "e301e8c2-2042-4641-a416-e26c636c2dfd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:09:05.475620 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:09:05.475595 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e301e8c2-2042-4641-a416-e26c636c2dfd-kube-api-access-z9c8q" pod "e301e8c2-2042-4641-a416-e26c636c2dfd" (UID: "e301e8c2-2042-4641-a416-e26c636c2dfd"). InnerVolumeSpecName "kube-api-access-z9c8q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:09:05.475752 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:09:05.475661 5437 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e301e8c2-2042-4641-a416-e26c636c2dfd-metrics-certs" pod "e301e8c2-2042-4641-a416-e26c636c2dfd" (UID: "e301e8c2-2042-4641-a416-e26c636c2dfd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:09:05.574788 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:09:05.574754 5437 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e301e8c2-2042-4641-a416-e26c636c2dfd-maas-api-tls\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:09:05.574788 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:09:05.574783 5437 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e301e8c2-2042-4641-a416-e26c636c2dfd-metrics-certs\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:09:05.574788 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:09:05.574793 5437 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9c8q\" (UniqueName: \"kubernetes.io/projected/e301e8c2-2042-4641-a416-e26c636c2dfd-kube-api-access-z9c8q\") on node \"ip-10-0-136-16.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:09:05.931417 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:09:05.931356 5437 generic.go:356] "Generic (PLEG): container finished" podID="e301e8c2-2042-4641-a416-e26c636c2dfd" containerID="9e0f91fdc0e28715bc88d22d937818a08b3a2a74ef18c7f12fe9f0fe1961a71e" exitCode=0 Nov 03 03:09:05.931630 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:09:05.931459 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6b6d569d44-rm7ch" event={"ID":"e301e8c2-2042-4641-a416-e26c636c2dfd","Type":"ContainerDied","Data":"9e0f91fdc0e28715bc88d22d937818a08b3a2a74ef18c7f12fe9f0fe1961a71e"} Nov 03 03:09:05.931630 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:09:05.931475 5437 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6b6d569d44-rm7ch" Nov 03 03:09:05.931630 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:09:05.931517 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6b6d569d44-rm7ch" event={"ID":"e301e8c2-2042-4641-a416-e26c636c2dfd","Type":"ContainerDied","Data":"505a023e5782077e5953c01b87a9f8dd9914f48b217cf24f16cafffc45319aa9"} Nov 03 03:09:05.931630 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:09:05.931548 5437 scope.go:122] "RemoveContainer" containerID="9e0f91fdc0e28715bc88d22d937818a08b3a2a74ef18c7f12fe9f0fe1961a71e" Nov 03 03:09:05.942982 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:09:05.942959 5437 scope.go:122] "RemoveContainer" containerID="9e0f91fdc0e28715bc88d22d937818a08b3a2a74ef18c7f12fe9f0fe1961a71e" Nov 03 03:09:05.943280 ip-10-0-136-16 kubenswrapper[5437]: E1103 03:09:05.943256 5437 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e0f91fdc0e28715bc88d22d937818a08b3a2a74ef18c7f12fe9f0fe1961a71e\": container with ID starting with 9e0f91fdc0e28715bc88d22d937818a08b3a2a74ef18c7f12fe9f0fe1961a71e not found: ID does not exist" containerID="9e0f91fdc0e28715bc88d22d937818a08b3a2a74ef18c7f12fe9f0fe1961a71e" Nov 03 03:09:05.943354 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:09:05.943291 5437 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e0f91fdc0e28715bc88d22d937818a08b3a2a74ef18c7f12fe9f0fe1961a71e"} err="failed to get container status \"9e0f91fdc0e28715bc88d22d937818a08b3a2a74ef18c7f12fe9f0fe1961a71e\": rpc error: code = NotFound desc = could not find container \"9e0f91fdc0e28715bc88d22d937818a08b3a2a74ef18c7f12fe9f0fe1961a71e\": container with ID starting with 9e0f91fdc0e28715bc88d22d937818a08b3a2a74ef18c7f12fe9f0fe1961a71e not found: ID does not exist" Nov 03 03:09:06.101525 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:09:06.101486 5437 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6b6d569d44-rm7ch"] Nov 03 03:09:06.105569 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:09:06.105543 5437 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6b6d569d44-rm7ch"] Nov 03 03:09:08.087577 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:09:08.087540 5437 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e301e8c2-2042-4641-a416-e26c636c2dfd" path="/var/lib/kubelet/pods/e301e8c2-2042-4641-a416-e26c636c2dfd/volumes" Nov 03 03:09:13.721085 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:09:13.721047 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 03:09:14.034514 ip-10-0-136-16 kubenswrapper[5437]: E1103 03:09:14.034420 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 03:09:18.671389 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:09:18.671317 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 03:09:44.034522 ip-10-0-136-16 kubenswrapper[5437]: E1103 03:09:44.034481 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 03:10:14.035180 ip-10-0-136-16 kubenswrapper[5437]: E1103 03:10:14.035097 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 03:10:14.122150 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:10:14.122101 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 03:10:19.067580 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:10:19.067538 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 03:10:24.338898 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:10:24.338865 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 03:10:24.545906 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:10:24.545863 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 03:10:44.033975 ip-10-0-136-16 kubenswrapper[5437]: E1103 03:10:44.033936 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 03:10:51.857765 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:10:51.857727 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 03:10:52.063828 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:10:52.063791 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 03:11:14.034192 ip-10-0-136-16 kubenswrapper[5437]: E1103 03:11:14.034155 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 03:11:14.445847 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:11:14.445809 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 03:11:17.536461 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:11:17.536204 5437 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-dbf96 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 03:11:17.537522 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:11:17.537483 5437 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dbf96" podUID="fbdf72a4-6fc0-4511-b73e-ceabc4a76c2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 03:11:19.402615 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:11:19.402579 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 03:11:44.037302 ip-10-0-136-16 kubenswrapper[5437]: E1103 03:11:44.037256 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 03:12:14.105860 ip-10-0-136-16 kubenswrapper[5437]: E1103 03:12:14.105815 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 03:12:14.841460 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:12:14.841412 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 03:12:19.779389 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:12:19.778925 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 03:12:44.041460 ip-10-0-136-16 kubenswrapper[5437]: E1103 03:12:44.041412 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 03:13:10.967659 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:13:10.967623 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 03:13:14.034310 ip-10-0-136-16 kubenswrapper[5437]: E1103 03:13:14.034274 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 03:13:15.164422 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:13:15.164319 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 03:13:20.102984 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:13:20.102946 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 03:13:29.183555 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:13:29.183501 5437 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-zj6mx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 03:13:29.184291 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:13:29.184249 5437 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-zj6mx" podUID="fd1010b6-bf5b-4633-a266-9b5dcf9b1963" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 03:13:44.033210 ip-10-0-136-16 kubenswrapper[5437]: E1103 03:13:44.033170 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 03:14:14.033917 ip-10-0-136-16 kubenswrapper[5437]: E1103 03:14:14.033874 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 03:14:15.536716 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:14:15.536678 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 03:14:20.503991 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:14:20.503772 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 03:14:32.533519 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:14:32.533470 5437 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-dbf96 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 03:14:32.534164 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:14:32.534128 5437 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dbf96" podUID="fbdf72a4-6fc0-4511-b73e-ceabc4a76c2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 03:14:42.879390 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:14:42.877496 5437 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-x2xwt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 03:14:42.879390 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:14:42.877584 5437 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-x2xwt" podUID="f9cb50c6-d76d-4138-8d00-6b7836297ed7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 03:14:44.034468 ip-10-0-136-16 kubenswrapper[5437]: E1103 03:14:44.034430 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 03:15:14.034100 ip-10-0-136-16 kubenswrapper[5437]: E1103 03:15:14.034062 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 03:15:15.878951 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:15:15.878914 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 03:15:20.853737 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:15:20.853699 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 03:15:37.897669 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:15:37.897633 5437 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-7b888454db-5q4k2_4de9a87f-21bb-4b1b-9700-0ceef4bbda2c/manager/0.log" Nov 03 03:15:38.763219 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:15:38.763177 5437 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-94pkh_a2288409-ad83-494d-8e5f-17cc3690265e/manager/0.log" Nov 03 03:15:39.622040 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:15:39.622003 5437 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-f48966749-2xjtr_ff7527e0-1356-406f-87b6-68f2e2f4e25d/manager/0.log" Nov 03 03:15:41.809527 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:15:41.809490 5437 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-67759c8968-ht5tg_1edae360-1d25-4785-b969-f27cf0fe5c5e/maas-api/0.log" Nov 03 03:15:44.033264 ip-10-0-136-16 kubenswrapper[5437]: E1103 03:15:44.033226 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 03:15:46.007485 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:15:46.007455 5437 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-529bb_096e923b-f4be-4e2b-913f-a2ecb011a0c0/postgres/0.log" Nov 03 03:15:48.772170 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:15:48.772082 5437 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-zlql6_1447e14e-f0a8-43bd-8e05-9d32b57fcf34/manager/0.log" Nov 03 03:15:51.996103 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:15:51.996062 5437 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-68dcf9949f-hcw7m_64c2563a-4b6a-4d94-8cb7-09fce6d2e734/kube-auth-proxy/0.log" Nov 03 03:15:53.307973 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:15:53.307935 5437 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-59f957fdd7-dbf96_fbdf72a4-6fc0-4511-b73e-ceabc4a76c2d/istio-proxy/0.log" Nov 03 03:15:54.171178 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:15:54.171142 5437 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-59f957fdd7-x2xwt_f9cb50c6-d76d-4138-8d00-6b7836297ed7/istio-proxy/0.log" Nov 03 03:15:54.716856 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:15:54.716823 5437 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-59f957fdd7-zj6mx_fd1010b6-bf5b-4633-a266-9b5dcf9b1963/istio-proxy/0.log" Nov 03 03:15:55.458540 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:15:55.458498 5437 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7987b45974-f58pp_9faced96-b336-439d-824a-2dd51fbf8e00/payload-processing/0.log" Nov 03 03:15:56.931225 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:15:56.931188 5437 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-5xgkr_e02c0d62-8c28-4b9c-9280-0843e225fe71/main/0.log" Nov 03 03:15:57.220456 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:15:57.220350 5437 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-qknt5_7b3bf52c-b366-481a-802f-4c2cc0384280/main/0.log" Nov 03 03:15:57.507886 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:15:57.507794 5437 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-rpmbj_9a552ecb-43a3-4154-b818-24e78b95b4f1/main/0.log" Nov 03 03:15:57.734850 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:15:57.734819 5437 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-jthts_7a292fcf-e87a-431c-81fe-33a292fc2d0a/main/0.log" Nov 03 03:15:57.960790 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:15:57.960754 5437 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qrnjn_835a3491-fa81-4354-924f-4d9ef651e58e/main/0.log" Nov 03 03:15:58.427548 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:15:58.427505 5437 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-h9gfn_d4d4e0c0-2ae5-402d-a47a-d8e4c79d7cfa/main/0.log" Nov 03 03:15:58.664400 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:15:58.664358 5437 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-l87z7_03a82c52-52db-44b7-bf71-64ec0c963054/main/0.log" Nov 03 03:16:09.376232 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:16:09.376195 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-zlr9k/must-gather-5rrnk"] Nov 03 03:16:09.376844 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:16:09.376820 5437 cpu_manager.go:414] "containerMap: removing container" podUID="e301e8c2-2042-4641-a416-e26c636c2dfd" containerName="maas-api" Nov 03 03:16:09.376936 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:16:09.376846 5437 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e301e8c2-2042-4641-a416-e26c636c2dfd" containerName="maas-api" Nov 03 03:16:09.377005 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:16:09.376969 5437 memory_manager.go:365] "RemoveStaleState removing state" podUID="e301e8c2-2042-4641-a416-e26c636c2dfd" containerName="maas-api" Nov 03 03:16:09.380078 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:16:09.380059 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zlr9k/must-gather-5rrnk" Nov 03 03:16:09.423687 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:16:09.423645 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 03:16:09.434062 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:16:09.434023 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zlr9k/must-gather-5rrnk"] Nov 03 03:16:09.456332 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:16:09.456302 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kg4tt\" (UniqueName: \"kubernetes.io/projected/72c35f5f-afe0-4829-a771-4a1a7d62f928-kube-api-access-kg4tt\") pod \"must-gather-5rrnk\" (UID: \"72c35f5f-afe0-4829-a771-4a1a7d62f928\") " pod="openshift-must-gather-zlr9k/must-gather-5rrnk" Nov 03 03:16:09.456520 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:16:09.456401 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/72c35f5f-afe0-4829-a771-4a1a7d62f928-must-gather-output\") pod \"must-gather-5rrnk\" (UID: \"72c35f5f-afe0-4829-a771-4a1a7d62f928\") " pod="openshift-must-gather-zlr9k/must-gather-5rrnk" Nov 03 03:16:09.483973 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:16:09.483941 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-zlr9k\"/\"default-dockercfg-8wmk6\"" Nov 03 03:16:09.484149 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:16:09.483973 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zlr9k\"/\"openshift-service-ca.crt\"" Nov 03 03:16:09.485608 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:16:09.485592 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zlr9k\"/\"kube-root-ca.crt\"" Nov 03 03:16:09.557080 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:16:09.557050 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kg4tt\" (UniqueName: \"kubernetes.io/projected/72c35f5f-afe0-4829-a771-4a1a7d62f928-kube-api-access-kg4tt\") pod \"must-gather-5rrnk\" (UID: \"72c35f5f-afe0-4829-a771-4a1a7d62f928\") " pod="openshift-must-gather-zlr9k/must-gather-5rrnk" Nov 03 03:16:09.557292 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:16:09.557109 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/72c35f5f-afe0-4829-a771-4a1a7d62f928-must-gather-output\") pod \"must-gather-5rrnk\" (UID: \"72c35f5f-afe0-4829-a771-4a1a7d62f928\") " pod="openshift-must-gather-zlr9k/must-gather-5rrnk" Nov 03 03:16:09.557492 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:16:09.557471 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/72c35f5f-afe0-4829-a771-4a1a7d62f928-must-gather-output\") pod \"must-gather-5rrnk\" (UID: \"72c35f5f-afe0-4829-a771-4a1a7d62f928\") " pod="openshift-must-gather-zlr9k/must-gather-5rrnk" Nov 03 03:16:09.629870 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:16:09.629787 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kg4tt\" (UniqueName: \"kubernetes.io/projected/72c35f5f-afe0-4829-a771-4a1a7d62f928-kube-api-access-kg4tt\") pod \"must-gather-5rrnk\" (UID: \"72c35f5f-afe0-4829-a771-4a1a7d62f928\") " pod="openshift-must-gather-zlr9k/must-gather-5rrnk" Nov 03 03:16:09.657216 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:16:09.657170 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 03:16:09.694763 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:16:09.694730 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zlr9k/must-gather-5rrnk" Nov 03 03:16:10.067717 ip-10-0-136-16 kubenswrapper[5437]: W1103 03:16:10.067679 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod72c35f5f_afe0_4829_a771_4a1a7d62f928.slice/crio-462d3c471c407d31657dc7a216739054a7e350dceafe18987102e8f3a7ef8ed4 WatchSource:0}: Error finding container 462d3c471c407d31657dc7a216739054a7e350dceafe18987102e8f3a7ef8ed4: Status 404 returned error can't find the container with id 462d3c471c407d31657dc7a216739054a7e350dceafe18987102e8f3a7ef8ed4 Nov 03 03:16:10.069546 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:16:10.069524 5437 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 03:16:10.069945 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:16:10.069923 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zlr9k/must-gather-5rrnk"] Nov 03 03:16:10.390214 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:16:10.390133 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zlr9k/must-gather-5rrnk" event={"ID":"72c35f5f-afe0-4829-a771-4a1a7d62f928","Type":"ContainerStarted","Data":"462d3c471c407d31657dc7a216739054a7e350dceafe18987102e8f3a7ef8ed4"} Nov 03 03:16:14.041201 ip-10-0-136-16 kubenswrapper[5437]: E1103 03:16:14.041158 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 03:16:16.225616 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:16:16.225575 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 03:16:18.444895 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:16:18.444855 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zlr9k/must-gather-5rrnk" event={"ID":"72c35f5f-afe0-4829-a771-4a1a7d62f928","Type":"ContainerStarted","Data":"eb7ec24769c384bf091983741f3f22eada402b2395bdb69cb7fc12617b98feac"} Nov 03 03:16:18.445492 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:16:18.445445 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zlr9k/must-gather-5rrnk" event={"ID":"72c35f5f-afe0-4829-a771-4a1a7d62f928","Type":"ContainerStarted","Data":"3f293888f26fa2f2e8240ac82b4177b1c6eac1e0ae78afd5126ca38036b91a87"} Nov 03 03:16:18.565487 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:16:18.565420 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-zlr9k/must-gather-5rrnk" podStartSLOduration=2.109650951 podStartE2EDuration="9.565405043s" totalImagesPullingTime="7.455754092s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:16:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:16:18.564013324 +0000 UTC m=+7060.593495619" watchObservedRunningTime="2026-11-03 03:16:18.565405043 +0000 UTC m=+7060.594887337" Nov 03 03:16:19.261460 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:16:19.261425 5437 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-dfr9t_6179266f-07ff-48cd-bf03-3f1e961529d7/global-pull-secret-syncer/0.log" Nov 03 03:16:21.259901 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:16:21.259863 5437 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" limit=1024 Nov 03 03:16:21.513048 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:16:21.512966 5437 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-7b888454db-5q4k2_4de9a87f-21bb-4b1b-9700-0ceef4bbda2c/manager/0.log" Nov 03 03:16:21.946713 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:16:21.946674 5437 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-dd9jt_fe434c5e-450b-4512-93aa-78d3e56e5002/konnectivity-agent/0.log" Nov 03 03:16:21.980261 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:16:21.980218 5437 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-94pkh_a2288409-ad83-494d-8e5f-17cc3690265e/manager/0.log" Nov 03 03:16:23.120994 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:16:23.120949 5437 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-f48966749-2xjtr_ff7527e0-1356-406f-87b6-68f2e2f4e25d/manager/0.log" Nov 03 03:16:23.246110 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:16:23.246074 5437 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-136-16.us-west-2.compute.internal_7f364a286e3c240d587086e0ec2180aa/haproxy/0.log" Nov 03 03:16:27.898971 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:16:27.898931 5437 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-7b888454db-5q4k2_4de9a87f-21bb-4b1b-9700-0ceef4bbda2c/manager/0.log" Nov 03 03:16:28.494132 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:16:28.494097 5437 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-94pkh_a2288409-ad83-494d-8e5f-17cc3690265e/manager/0.log" Nov 03 03:16:29.431482 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:16:29.431432 5437 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-kjgqj/perf-node-gather-daemonset-jcjck"] Nov 03 03:16:29.449494 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:16:29.449445 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kjgqj/perf-node-gather-daemonset-jcjck" Nov 03 03:16:29.452108 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:16:29.452075 5437 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-f48966749-2xjtr_ff7527e0-1356-406f-87b6-68f2e2f4e25d/manager/0.log" Nov 03 03:16:29.521208 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:16:29.521167 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kjgqj/perf-node-gather-daemonset-jcjck"] Nov 03 03:16:29.558477 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:16:29.558426 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/aea7b199-e6c1-4cb7-afd5-729cd29f7877-proc\") pod \"perf-node-gather-daemonset-jcjck\" (UID: \"aea7b199-e6c1-4cb7-afd5-729cd29f7877\") " pod="openshift-must-gather-kjgqj/perf-node-gather-daemonset-jcjck" Nov 03 03:16:29.558477 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:16:29.558472 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/aea7b199-e6c1-4cb7-afd5-729cd29f7877-lib-modules\") pod \"perf-node-gather-daemonset-jcjck\" (UID: \"aea7b199-e6c1-4cb7-afd5-729cd29f7877\") " pod="openshift-must-gather-kjgqj/perf-node-gather-daemonset-jcjck" Nov 03 03:16:29.558709 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:16:29.558573 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/aea7b199-e6c1-4cb7-afd5-729cd29f7877-sys\") pod \"perf-node-gather-daemonset-jcjck\" (UID: \"aea7b199-e6c1-4cb7-afd5-729cd29f7877\") " pod="openshift-must-gather-kjgqj/perf-node-gather-daemonset-jcjck" Nov 03 03:16:29.558709 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:16:29.558648 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktxtk\" (UniqueName: \"kubernetes.io/projected/aea7b199-e6c1-4cb7-afd5-729cd29f7877-kube-api-access-ktxtk\") pod \"perf-node-gather-daemonset-jcjck\" (UID: \"aea7b199-e6c1-4cb7-afd5-729cd29f7877\") " pod="openshift-must-gather-kjgqj/perf-node-gather-daemonset-jcjck" Nov 03 03:16:29.558786 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:16:29.558711 5437 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/aea7b199-e6c1-4cb7-afd5-729cd29f7877-podres\") pod \"perf-node-gather-daemonset-jcjck\" (UID: \"aea7b199-e6c1-4cb7-afd5-729cd29f7877\") " pod="openshift-must-gather-kjgqj/perf-node-gather-daemonset-jcjck" Nov 03 03:16:29.574159 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:16:29.574122 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kjgqj\"/\"openshift-service-ca.crt\"" Nov 03 03:16:29.574329 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:16:29.574218 5437 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kjgqj\"/\"kube-root-ca.crt\"" Nov 03 03:16:29.619074 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:16:29.619040 5437 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-kjgqj\"/\"default-dockercfg-9859t\"" Nov 03 03:16:29.660214 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:16:29.660178 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/aea7b199-e6c1-4cb7-afd5-729cd29f7877-proc\") pod \"perf-node-gather-daemonset-jcjck\" (UID: \"aea7b199-e6c1-4cb7-afd5-729cd29f7877\") " pod="openshift-must-gather-kjgqj/perf-node-gather-daemonset-jcjck" Nov 03 03:16:29.660214 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:16:29.660217 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/aea7b199-e6c1-4cb7-afd5-729cd29f7877-lib-modules\") pod \"perf-node-gather-daemonset-jcjck\" (UID: \"aea7b199-e6c1-4cb7-afd5-729cd29f7877\") " pod="openshift-must-gather-kjgqj/perf-node-gather-daemonset-jcjck" Nov 03 03:16:29.660507 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:16:29.660265 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/aea7b199-e6c1-4cb7-afd5-729cd29f7877-sys\") pod \"perf-node-gather-daemonset-jcjck\" (UID: \"aea7b199-e6c1-4cb7-afd5-729cd29f7877\") " pod="openshift-must-gather-kjgqj/perf-node-gather-daemonset-jcjck" Nov 03 03:16:29.660507 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:16:29.660326 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/aea7b199-e6c1-4cb7-afd5-729cd29f7877-proc\") pod \"perf-node-gather-daemonset-jcjck\" (UID: \"aea7b199-e6c1-4cb7-afd5-729cd29f7877\") " pod="openshift-must-gather-kjgqj/perf-node-gather-daemonset-jcjck" Nov 03 03:16:29.660507 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:16:29.660352 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktxtk\" (UniqueName: \"kubernetes.io/projected/aea7b199-e6c1-4cb7-afd5-729cd29f7877-kube-api-access-ktxtk\") pod \"perf-node-gather-daemonset-jcjck\" (UID: \"aea7b199-e6c1-4cb7-afd5-729cd29f7877\") " pod="openshift-must-gather-kjgqj/perf-node-gather-daemonset-jcjck" Nov 03 03:16:29.660507 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:16:29.660409 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/aea7b199-e6c1-4cb7-afd5-729cd29f7877-sys\") pod \"perf-node-gather-daemonset-jcjck\" (UID: \"aea7b199-e6c1-4cb7-afd5-729cd29f7877\") " pod="openshift-must-gather-kjgqj/perf-node-gather-daemonset-jcjck" Nov 03 03:16:29.660507 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:16:29.660450 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/aea7b199-e6c1-4cb7-afd5-729cd29f7877-lib-modules\") pod \"perf-node-gather-daemonset-jcjck\" (UID: \"aea7b199-e6c1-4cb7-afd5-729cd29f7877\") " pod="openshift-must-gather-kjgqj/perf-node-gather-daemonset-jcjck" Nov 03 03:16:29.660507 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:16:29.660468 5437 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/aea7b199-e6c1-4cb7-afd5-729cd29f7877-podres\") pod \"perf-node-gather-daemonset-jcjck\" (UID: \"aea7b199-e6c1-4cb7-afd5-729cd29f7877\") " pod="openshift-must-gather-kjgqj/perf-node-gather-daemonset-jcjck" Nov 03 03:16:29.660807 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:16:29.660562 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/aea7b199-e6c1-4cb7-afd5-729cd29f7877-podres\") pod \"perf-node-gather-daemonset-jcjck\" (UID: \"aea7b199-e6c1-4cb7-afd5-729cd29f7877\") " pod="openshift-must-gather-kjgqj/perf-node-gather-daemonset-jcjck" Nov 03 03:16:29.723801 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:16:29.723719 5437 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktxtk\" (UniqueName: \"kubernetes.io/projected/aea7b199-e6c1-4cb7-afd5-729cd29f7877-kube-api-access-ktxtk\") pod \"perf-node-gather-daemonset-jcjck\" (UID: \"aea7b199-e6c1-4cb7-afd5-729cd29f7877\") " pod="openshift-must-gather-kjgqj/perf-node-gather-daemonset-jcjck" Nov 03 03:16:29.765928 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:16:29.765884 5437 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kjgqj/perf-node-gather-daemonset-jcjck" Nov 03 03:16:29.781397 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:16:29.781349 5437 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-zlql6_1447e14e-f0a8-43bd-8e05-9d32b57fcf34/manager/0.log" Nov 03 03:16:30.140085 ip-10-0-136-16 kubenswrapper[5437]: W1103 03:16:30.140037 5437 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podaea7b199_e6c1_4cb7_afd5_729cd29f7877.slice/crio-bc193b40719af58da1340d3f6b0164fd5c7cb3f43806c0fad47543c5c0a28ae9 WatchSource:0}: Error finding container bc193b40719af58da1340d3f6b0164fd5c7cb3f43806c0fad47543c5c0a28ae9: Status 404 returned error can't find the container with id bc193b40719af58da1340d3f6b0164fd5c7cb3f43806c0fad47543c5c0a28ae9 Nov 03 03:16:30.141791 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:16:30.141760 5437 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kjgqj/perf-node-gather-daemonset-jcjck"] Nov 03 03:16:30.507777 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:16:30.507719 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kjgqj/perf-node-gather-daemonset-jcjck" event={"ID":"aea7b199-e6c1-4cb7-afd5-729cd29f7877","Type":"ContainerStarted","Data":"c198d4d540f2d0eae2a1bebd765650b910dc81638671b002c404067e6217d77b"} Nov 03 03:16:30.508212 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:16:30.507798 5437 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kjgqj/perf-node-gather-daemonset-jcjck" event={"ID":"aea7b199-e6c1-4cb7-afd5-729cd29f7877","Type":"ContainerStarted","Data":"bc193b40719af58da1340d3f6b0164fd5c7cb3f43806c0fad47543c5c0a28ae9"} Nov 03 03:16:30.508212 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:16:30.507981 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-kjgqj/perf-node-gather-daemonset-jcjck" Nov 03 03:16:30.628724 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:16:30.628662 5437 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-kjgqj/perf-node-gather-daemonset-jcjck" podStartSLOduration=1.628641236 podStartE2EDuration="1.628641236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 03:16:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:16:30.626874237 +0000 UTC m=+7072.656356531" watchObservedRunningTime="2026-11-03 03:16:30.628641236 +0000 UTC m=+7072.658123582" Nov 03 03:16:34.224091 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:16:34.224049 5437 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-7b888454db-5q4k2_4de9a87f-21bb-4b1b-9700-0ceef4bbda2c/manager/0.log" Nov 03 03:16:34.672553 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:16:34.672509 5437 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-94pkh_a2288409-ad83-494d-8e5f-17cc3690265e/manager/0.log" Nov 03 03:16:35.705435 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:16:35.705390 5437 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-f48966749-2xjtr_ff7527e0-1356-406f-87b6-68f2e2f4e25d/manager/0.log" Nov 03 03:16:36.526050 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:16:36.526013 5437 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-kjgqj/perf-node-gather-daemonset-jcjck" Nov 03 03:16:37.968422 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:16:37.968381 5437 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-htfx6_766a044f-fc66-4998-8006-343c0329860d/kube-state-metrics/0.log" Nov 03 03:16:38.393849 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:16:38.393810 5437 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-htfx6_766a044f-fc66-4998-8006-343c0329860d/kube-rbac-proxy-main/0.log" Nov 03 03:16:38.822589 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:16:38.822548 5437 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-htfx6_766a044f-fc66-4998-8006-343c0329860d/kube-rbac-proxy-self/0.log" Nov 03 03:16:39.688642 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:16:39.688612 5437 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-ssqxs_9eb34322-124e-484d-a4fa-d79391ac36b1/monitoring-plugin/0.log" Nov 03 03:16:40.123053 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:16:40.123015 5437 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5z5ct_30eede54-1970-4f52-b219-70d3c469ddd4/node-exporter/0.log" Nov 03 03:16:40.552242 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:16:40.552205 5437 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5z5ct_30eede54-1970-4f52-b219-70d3c469ddd4/kube-rbac-proxy/0.log" Nov 03 03:16:40.989729 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:16:40.989652 5437 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5z5ct_30eede54-1970-4f52-b219-70d3c469ddd4/init-textfile/0.log" Nov 03 03:16:43.658800 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:16:43.658765 5437 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-68dcf9949f-hcw7m_64c2563a-4b6a-4d94-8cb7-09fce6d2e734/kube-auth-proxy/0.log" Nov 03 03:16:44.004061 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:16:44.003962 5437 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-lcwkf_0e35c4cd-ef18-40c1-8568-f77773e378b2/kube-rbac-proxy-main/0.log" Nov 03 03:16:44.037277 ip-10-0-136-16 kubenswrapper[5437]: E1103 03:16:44.037236 5437 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bbfdw" containerName="extractor" Nov 03 03:16:44.431761 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:16:44.431722 5437 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-lcwkf_0e35c4cd-ef18-40c1-8568-f77773e378b2/kube-rbac-proxy-self/0.log" Nov 03 03:16:44.769154 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:16:44.769071 5437 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-59f957fdd7-dbf96_fbdf72a4-6fc0-4511-b73e-ceabc4a76c2d/istio-proxy/0.log" Nov 03 03:16:44.861724 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:16:44.861680 5437 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-lcwkf_0e35c4cd-ef18-40c1-8568-f77773e378b2/openshift-state-metrics/0.log" Nov 03 03:16:45.456930 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:16:45.456891 5437 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-59f957fdd7-x2xwt_f9cb50c6-d76d-4138-8d00-6b7836297ed7/istio-proxy/0.log" Nov 03 03:16:45.892126 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:16:45.892086 5437 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-59f957fdd7-zj6mx_fd1010b6-bf5b-4633-a266-9b5dcf9b1963/istio-proxy/0.log" Nov 03 03:16:46.547804 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:16:46.547771 5437 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7987b45974-f58pp_9faced96-b336-439d-824a-2dd51fbf8e00/payload-processing/0.log" Nov 03 03:16:50.807942 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:16:50.807905 5437 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-8bbf59f7f-cmcdf_8f478179-5984-48dd-b872-8f552954ecea/thanos-query/0.log" Nov 03 03:16:51.237297 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:16:51.237206 5437 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-8bbf59f7f-cmcdf_8f478179-5984-48dd-b872-8f552954ecea/kube-rbac-proxy-web/0.log" Nov 03 03:16:51.670608 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:16:51.670566 5437 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-8bbf59f7f-cmcdf_8f478179-5984-48dd-b872-8f552954ecea/kube-rbac-proxy/0.log" Nov 03 03:16:52.103998 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:16:52.103967 5437 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-8bbf59f7f-cmcdf_8f478179-5984-48dd-b872-8f552954ecea/prom-label-proxy/0.log" Nov 03 03:16:52.580555 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:16:52.580519 5437 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-zlql6_1447e14e-f0a8-43bd-8e05-9d32b57fcf34/manager/0.log" Nov 03 03:16:52.666571 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:16:52.666537 5437 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-8bbf59f7f-cmcdf_8f478179-5984-48dd-b872-8f552954ecea/kube-rbac-proxy-rules/0.log" Nov 03 03:16:53.094276 ip-10-0-136-16 kubenswrapper[5437]: I1103 03:16:53.094232 5437 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-8bbf59f7f-cmcdf_8f478179-5984-48dd-b872-8f552954ecea/kube-rbac-proxy-metrics/0.log"